Three-dimensional image display method, three-dimensional image display device, and recording medium

ABSTRACT

In a three-dimensional image display method, a processor acquires first three-dimensional data and second three-dimensional data of a subject from a recording medium. The processor converts a first three-dimensional coordinate system of the first three-dimensional data and a second three-dimensional coordinate system of the second three-dimensional data into a three-dimensional common coordinate system on the basis of structure information related to a geometric structure of the subject. The processor displays an image of the first three-dimensional data in the common coordinate system and an image of the second three-dimensional data in the common coordinate system on a display.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a three-dimensional image displaymethod, a three-dimensional image display device, and a recordingmedium.

Priority is claimed on Japanese Patent Application No. 2021-092249,filed on Jun. 1, 2021, the content of which is incorporated herein byreference.

Description of Related Art

Industrial endoscope devices have been used for an inspection ofinternal abnormalities (damage, corrosion, and the like) of boilers, gasturbines, automobile engines, pipes, and the like. In an endoscopicinspection, an inspection worker records a still image during theinspection in order to record whether an abnormality has occurred orrecord severity of an abnormality as proof of the inspection. After theendoscopic inspection is completed, an inspection report is generated.In general, text indicating the state or the like of an abnormality seenin a recorded still image is attached to the inspection report alongwith the still image.

Position information is included in an additional item related to anabnormality. The position information indicates a position at which arecorded still image is acquired in an inspection target. The positioninformation of a found abnormality is important when the abnormalportion is replaced or fixed, or when a next inspection is performed. Amethod of recording and managing position information of an abnormalityis disclosed. The method associates a still image recorded during aninspection with a specific position in three-dimensional data (3D data)indicating a three-dimensional shape (3D shape) of an inspection targetand visualizes a position at which the still image is acquired. By usingthis method, the position of the inspection target in which the stillimage is acquired becomes clear.

A method of acquiring 3D data of an inspection target is disclosed. Themethod reconfigures a 3D shape of the inspection target by using a videorecorded during an inspection. In this method, a video recorded duringthe inspection is used. Therefore, special inspection equipment does notneed to be brought in, and a blueprint or the like of the inspectiontarget is unnecessary.

In the method of reconfiguring the 3D shape of the inspection target, aplurality of images acquired at a plurality of viewpoints need to beassociated with each other. If it is difficult to associate theplurality of images with each other, there is a problem in that 3D dataare divided into multiple pieces of data. For example, there is a casein which a distal end of an endoscope suddenly moves in the process ofrecording a video and the composition of an image greatly changes.Alternatively, there is a case in which halation or the like occurs inan image in the process of recording a video and the state of the imagegreatly changes. Alternatively, there is a case in which recording of avideo is resumed after interruption and an inspection target seen in animage acquired after the resumption does not match an inspection targetseen in an image acquired before the interruption. In these cases, 3Ddata are divided into multiple pieces of data.

It is preferable for a user that 3D data of the widest possible regionof an inspection target be configured as a single piece of data withoutbeing divided. If multiple pieces of 3D data of the inspection targetare generated, it is important to connect the multiple pieces of 3D datatogether and generate 3D data of a wide region of the inspection target.The following technique is disclosed as a method of connecting multiplepieces of 3D data of a partial region together so as to generate 3D dataof a wide region.

A technique disclosed in Japanese Patent No. 6040882 provides a methodof connecting multiple pieces of 3D data together by using arelationship between the 3D data and a two-dimensional image group usedfor generating the 3D data. First 3D data are associated with a firsttwo-dimensional image group, and second 3D data are associated with asecond two-dimensional image group. A first image and a second imageincluding the same feature point are selected. The first image isincluded in the first two-dimensional image group, and the second imageis included in the second two-dimensional image group. Translation androtation of 3D data are performed such that the three-dimensionalcoordinates of the feature point in the first image match thethree-dimensional coordinates of the feature point in the second image.Thereafter, errors of the three-dimensional coordinates are minimized.

A technique disclosed in each of Japanese Patent No. 6811763 andJapanese Unexamined Patent Application, First Publication No.2010-066595 provides a method of connecting multiple pieces of 3D datatogether on the basis of a 3D shape indicated by the 3D data. In thetechnique disclosed in Japanese Patent No. 6811763, a feature ofgrounds, a feature of planes, or a feature of pillars is allocated tomultiple pieces of 3D data of a partial region. The multiple pieces of3D data are connected together such that the features of one piece ofthe 3D data match the features of another piece of the 3D data. In thetechnique disclosed in Japanese Unexamined Patent Application, FirstPublication No. 2010-066595, three planes are detected in each of themultiple pieces of 3D data. Normal lines of three planes are orthogonalto each other. The multiple pieces of 3D data are connected togethersuch that the three planes of one piece of the 3D data match the threeplanes of another piece of the 3D data.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, athree-dimensional image display method includes a first acquisitionstep, a second acquisition step, a conversion step, and a display step.A processor connects to a recording medium storing firstthree-dimensional data of a subject and second three-dimensional data ofthe subject and acquires the first three-dimensional data from therecording medium in the first acquisition step. The firstthree-dimensional data include three-dimensional coordinates defined ina first three-dimensional coordinate system. The secondthree-dimensional data include three-dimensional coordinates defined ina second three-dimensional coordinate system different from the firstthree-dimensional coordinate system. At least part of a region of thesubject corresponding to the first three-dimensional data is differentfrom at least part of a region of the subject corresponding to thesecond three-dimensional data. The processor connects to the recordingmedium and acquires the second three-dimensional data from the recordingmedium in the second acquisition step. The processor converts the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into a three-dimensional common coordinate system onthe basis of structure information related to a geometric structure ofthe subject in the conversion step. The structure information isgenerated without using the first three-dimensional data or the secondthree-dimensional data. The processor displays an image of the firstthree-dimensional data in the common coordinate system and an image ofthe second three-dimensional data in the common coordinate system on adisplay in the display step.

According to a second aspect of the present invention, in the firstaspect, the first three-dimensional data may be generated by using twoor more first images acquired at two or more different viewpoints. Thesecond three-dimensional data may be generated by using two or moresecond images acquired at two or more different viewpoints. At least oneof the two or more first images and at least one of the two or moresecond images may be different from each other.

According to a third aspect of the present invention, in the secondaspect, each of the two or more first images and each of the two or moresecond images may include time information. The processor may convertthe first three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system onthe basis of the time information in the conversion step.

According to a fourth aspect of the present invention, in the secondaspect, the two or more first images may be included in two or moreimages included in a first video. The two or more second images may beincluded in two or more images included in a second video that is thesame as or different from the first video.

According to a fifth aspect of the present invention, in the fourthaspect, the two or more first images and the two or more second imagesmay be included in the same video file.

According to a sixth aspect of the present invention, in the fifthaspect, the three-dimensional image display method may further include acalculation step in which the processor calculates a position of a lostregion on the basis of the number of the two or more first images, thenumber of the two or more second images, and the number of third images.The third images are temporally disposed between a set of the two ormore first images and a set of the two or more second images in thevideo file. The lost region is a region of the subject different fromany one of a first region of the subject and a second region of thesubject. The first region corresponds to the three-dimensionalcoordinates included in the first three-dimensional data. The secondregion corresponds to the three-dimensional coordinates included in thesecond three-dimensional data. The processor may convert the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system on the basis of theposition of the lost region in the conversion step.

According to a seventh aspect of the present invention, in the sixthaspect, the processor may calculate a shape of the lost region in thecalculation step. The processor may convert the first three-dimensionalcoordinate system and the second three-dimensional coordinate systeminto the common coordinate system on the basis of the shape of the lostregion in the conversion step.

According to an eighth aspect of the present invention, in the secondaspect, the structure information may indicate two or more positions atwhich a distal end of a movable insertion unit capable of being insertedinside an object having the subject is sequentially disposed.

According to a ninth aspect of the present invention, in the eighthaspect, the structure information may include first position informationand second position information. The first position informationindicates two or more positions at which the distal end is sequentiallydisposed in order to acquire the two or more first images. The secondposition information indicates two or more positions at which the distalend is sequentially disposed in order to acquire the two or more secondimages. The three-dimensional image display method may further include ageneration step in which the processor generates a position conversionparameter and a posture conversion parameter used for converting thefirst three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system onthe basis of the first position information and the second positioninformation. The processor may convert the first three-dimensionalcoordinate system and the second three-dimensional coordinate systeminto the common coordinate system by using the position conversionparameter and the posture conversion parameter in the conversion step.

According to a tenth aspect of the present invention, in the secondaspect, the processor may convert the first three-dimensional coordinatesystem and the second three-dimensional coordinate system into thecommon coordinate system such that a first region of the subject and asecond region of the subject are connected together in the conversionstep. The first region corresponds to the three-dimensional coordinatesincluded in the first three-dimensional data. The second regioncorresponds to the three-dimensional coordinates included in the secondthree-dimensional data. The processor may display information indicatingpositions of the first region and the second region on the display inthe display step.

According to an eleventh aspect of the present invention, in the tenthaspect, the processor may display information indicating accuracy ofconnection between the first region and the second region on the displayin the display step.

According to a twelfth aspect of the present invention, in the secondaspect, the two or more first images and the two or more second imagesmay be generated by an endoscope.

According to a thirteenth aspect of the present invention, in the secondaspect, the two or more first images and the two or more second imagesmay be generated on the basis of an optical image of the subjectacquired through a single-eye optical system.

According to a fourteenth aspect of the present invention, in the secondaspect, the three-dimensional image display method may further include ageneration step in which the processor generates a scale conversionparameter used for correcting at least one of a scale of athree-dimensional shape indicated by the first three-dimensional dataand a scale of a three-dimensional shape indicated by the secondthree-dimensional data. The processor may convert the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system by using the scaleconversion parameter in the conversion step.

According to a fifteenth aspect of the present invention, in the secondaspect, the structure information may be configured as design dataincluding a design value of the geometric structure or is configured asthree-dimensional data different from any of the first three-dimensionaldata and the second three-dimensional data.

According to a sixteenth aspect of the present invention, in the secondaspect, the structure information may be generated on the basis of dataoutput from a sensor.

According to a seventeenth aspect of the present invention, in thesecond aspect, the two or more first images and the two or more secondimages may be generated on the basis of an optical image of the subjectacquired by an insertion unit. The insertion unit may be capable ofbeing inserted inside an object having the subject and may be bendable.The structure information may be generated on the basis of informationindicating a bending direction and a bending amount of the insertionunit.

According to an eighteenth aspect of the present invention, in thesecond aspect, the three-dimensional image display method may furtherinclude a generation step in which the processor generates a positionconversion parameter and a posture conversion parameter used forconverting the first three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system onthe basis of the structure information. The processor may convert thefirst three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system byusing the position conversion parameter and the posture conversionparameter in the conversion step.

According to a nineteenth aspect of the present invention, in the secondaspect, the three-dimensional image display method may further includean adjustment step in which the processor adjusts at least one of aposition and a posture of at least one of the first three-dimensionaldata and the second three-dimensional data in the common coordinatesystem.

According to a twentieth aspect of the present invention, in the secondaspect, the subject may include two or more objects. The structureinformation may indicate positions at which the two or more objects aredisposed.

According to a twenty-first aspect of the present invention, athree-dimensional image display device includes a processor. Theprocessor is configured to connect to a recording medium storing firstthree-dimensional data of a subject and second three-dimensional data ofthe subject. The first three-dimensional data include three-dimensionalcoordinates defined in a first three-dimensional coordinate system. Thesecond three-dimensional data include three-dimensional coordinatesdefined in a second three-dimensional coordinate system different fromthe first three-dimensional coordinate system. At least part of a regionof the subject corresponding to the first three-dimensional data isdifferent from at least part of a region of the subject corresponding tothe second three-dimensional data. The processor is configured toacquire the first three-dimensional data and the secondthree-dimensional data from the recording medium. The processor isconfigured to convert the first three-dimensional coordinate system andthe second three-dimensional coordinate system into a three-dimensionalcommon coordinate system on the basis of structure information relatedto a geometric structure of the subject. The structure information isgenerated without using the first three-dimensional data or the secondthree-dimensional data. The processor is configured to display an imageof the first three-dimensional data in the common coordinate system andan image of the second three-dimensional data in the common coordinatesystem on a display.

According to a twenty-second aspect of the present invention, anon-transitory computer-readable recording medium stores a programcausing a computer to execute a first acquisition step, a secondacquisition step, a conversion step, and a display step. The computerconnects to a recording medium storing first three-dimensional data of asubject and second three-dimensional data of the subject and acquiresthe first three-dimensional data from the recording medium in the firstacquisition step. The first three-dimensional data includethree-dimensional coordinates defined in a first three-dimensionalcoordinate system. The second three-dimensional data includethree-dimensional coordinates defined in a second three-dimensionalcoordinate system different from the first three-dimensional coordinatesystem. At least part of a region of the subject corresponding to thefirst three-dimensional data is different from at least part of a regionof the subject corresponding to the second three-dimensional data. Thecomputer connects to the recording medium and acquires the secondthree-dimensional data from the recording medium in the secondacquisition step. The computer converts the first three-dimensionalcoordinate system and the second three-dimensional coordinate systeminto a three-dimensional common coordinate system on the basis ofstructure information related to a geometric structure of the subject inthe conversion step. The structure information is generated withoutusing the first three-dimensional data or the second three-dimensionaldata. The computer displays an image of the first three-dimensional datain the common coordinate system and an image of the secondthree-dimensional data in the common coordinate system on a display inthe display step.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of an image displaydevice according to a first embodiment of the present invention.

FIG. 2 is a flow chart showing a procedure of processing executed by theimage display device according to the first embodiment of the presentinvention.

FIG. 3 is a diagram showing an example of an image displayed on adisplay unit according to the first embodiment of the present invention.

FIG. 4 is a diagram showing an example of a dialog box displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 5 is a diagram showing an example of a reference table in the firstembodiment of the present invention.

FIG. 6 is a flow chart showing a procedure of processing executed by theimage display device according to the first embodiment of the presentinvention.

FIG. 7 is a diagram showing an example of an image displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 8 is a diagram showing an example of an image displayed on thedisplay unit according to the first embodiment of the present invention.

FIG. 9 is a block diagram showing a configuration of an image displaydevice according to a first modified example of the first embodiment ofthe present invention.

FIG. 10 is a flow chart showing a procedure of processing executed bythe image display device according to the first modified example of thefirst embodiment of the present invention.

FIG. 11 is a diagram showing an example of an image displayed on thedisplay unit according to the first modified example of the firstembodiment of the present invention.

FIG. 12 is a diagram showing an example of an image displayed on adisplay unit according to a second modified example of the firstembodiment of the present invention.

FIG. 13 is a diagram showing an example of an image displayed on thedisplay unit according to the second modified example of the firstembodiment of the present invention.

FIG. 14 is a diagram showing an example of a dialog box displayed on adisplay unit according to a third modified example of the firstembodiment of the present invention.

FIG. 15 is a diagram showing an example of an image displayed on thedisplay unit according to the third modified example of the firstembodiment of the present invention.

FIG. 16 is a diagram showing an example of a U-shaped tube according toa fourth modified example of the first embodiment of the presentinvention.

FIG. 17 is a diagram showing an example of an image displayed on adisplay unit according to the fourth modified example of the firstembodiment of the present invention.

FIG. 18 is a block diagram showing a configuration of blades in a fifthmodified example of the first embodiment of the present invention.

FIG. 19 is a flow chart showing a procedure of processing executed by animage display device according to the fifth modified example of thefirst embodiment of the present invention.

FIG. 20 is a diagram showing an example of an image displayed on adisplay unit according to the fifth modified example of the firstembodiment of the present invention.

FIG. 21 is a diagram showing an example of an image displayed on thedisplay unit according to the fifth modified example of the firstembodiment of the present invention.

FIG. 22 is a perspective view showing an entire configuration of anendoscope device according to a second embodiment of the presentinvention.

FIG. 23 is a block diagram showing an internal configuration of theendoscope device according to the second embodiment of the presentinvention.

FIG. 24 is a block diagram showing a functional configuration of a CPUincluded in the endoscope device according to the second embodiment ofthe present invention.

FIG. 25 is a block diagram showing a functional configuration of a datageneration unit included in the endoscope device according to the secondembodiment of the present invention.

FIG. 26 is a schematic diagram showing a situation in which an image isacquired in the second embodiment of the present invention.

FIG. 27 is a flow chart showing a procedure of processing for generating3D data in the second embodiment of the present invention.

FIG. 28 is a block diagram showing a configuration of an image displaydevice according to a third embodiment of the present invention.

FIG. 29 is a flow chart showing a procedure of processing executed bythe image display device according to the third embodiment of thepresent invention.

FIG. 30 is a flow chart showing a procedure of processing executed by animage display device according to a fourth embodiment of the presentinvention.

FIG. 31 is a diagram showing an example of an image displayed on adisplay unit according to the fourth embodiment of the presentinvention.

FIG. 32 is a diagram showing an example of an image displayed on thedisplay unit according to the fourth embodiment of the presentinvention.

FIG. 33 is a diagram showing reference data in a first modified exampleof the fourth embodiment of the present invention.

FIG. 34 is a flow chart showing a procedure of processing executed by animage display device according to the first modified example of thefourth embodiment of the present invention.

FIG. 35 is a diagram showing an example of an image displayed on adisplay unit according to the first modified example of the fourthembodiment of the present invention.

FIG. 36 is a diagram showing a structure of a combustion chamber in asecond modified example of the fourth embodiment of the presentinvention.

FIG. 37 is a diagram showing an example of an image displayed on adisplay unit according to the second modified example of the fourthembodiment of the present invention.

FIG. 38 is a block diagram showing a configuration of an image displaydevice according to a fifth embodiment of the present invention.

FIG. 39 is a flow chart showing a procedure of processing executed bythe image display device according to the fifth embodiment of thepresent invention.

FIG. 40 is a diagram showing an example of an image displayed on adisplay unit according to the fifth embodiment of the present invention.

FIG. 41 is a diagram showing an example of an image displayed on thedisplay unit according to the fifth embodiment of the present invention.

FIG. 42 is a diagram showing an example of an image displayed on thedisplay unit according to the fifth embodiment of the present invention.

FIG. 43 is a diagram showing an example of an image displayed on thedisplay unit according to the fifth embodiment of the present invention.

FIG. 44 is a diagram showing a method of calculating a lost section inthe fifth embodiment of the present invention.

FIG. 45 is a diagram showing a relationship between camera trace dataand 3D data in the fifth embodiment of the present invention.

FIG. 46 is a diagram showing a video file in the fifth embodiment of thepresent invention.

FIG. 47 is a flow chart showing a procedure of processing executed by animage display device according to a modified example of the fifthembodiment of the present invention.

FIG. 48 is a diagram showing an example of an image displayed on adisplay unit according to the modified example of the fifth embodimentof the present invention.

FIG. 49 is a flow chart showing a procedure of processing executed by animage display device according to a sixth embodiment of the presentinvention.

FIG. 50 is a flow chart showing a procedure of processing executed bythe image display device according to the sixth embodiment of thepresent invention.

FIG. 51 is a diagram showing an example of an image displayed on adisplay unit according to the sixth embodiment of the present invention.

FIG. 52 is a diagram showing an example of an image displayed on thedisplay unit according to the sixth embodiment of the present invention.

FIG. 53 is a diagram showing a structure of a heat exchanger in thesixth embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

First Embodiment

FIG. 1 shows a configuration of an image display device 50 according toa first embodiment of the present invention. The image display device 50shown in FIG. 1 includes a control unit 51, a data acquisition unit 52,a parameter generation unit 53, a conversion unit 54, a data generationunit 55, a display control unit 56, an information acceptance unit 57,and a structure estimation unit 58. An operation unit 70, a display unit71, a communication unit 72, and a storage unit 73 shown in FIG. 1 areconnected to the image display device 50. The image display device 50may include at least one of the operation unit 70, the display unit 71,the communication unit 72, and the storage unit 73.

For example, the image display device 50 is a personal computer (PC).The image display device 50 may be any one of a desktop PC, a laptop PC,and a tablet terminal. The image display device 50 may be a computersystem that operates on a cloud.

The operation unit 70 is a user interface. For example, the operationunit 70 is at least one of a button, a switch, a key, a mouse, ajoystick, a touch pad, a track ball, and a touch panel. The operationunit 70 accepts an operation from a user. A user can input various kindsof information into the image display device 50 by operating theoperation unit 70. The operation unit 70 accepts information input bythe user and outputs the information to the image display device 50.

The display unit 71 includes a display screen and displays an image orthe like of 3D data on the display screen. The display unit 71 is amonitor (display) such as a liquid crystal display (LCD). The displayunit 71 may be a touch panel. In such a case, the operation unit 70 andthe display unit 71 are integrated.

The communication unit 72 performs communication with an external devicesuch as an endoscope device. For example, the communication unit 72 isconnected to the external device wirelessly or by a cable. Thecommunication between the communication unit 72 and the external devicemay be performed via a local area network (LAN) or the Internet.

The storage unit 73 is a nonvolatile memory. For example, the storageunit 73 is at least one of a static random-access memory (SRAM), aread-only memory (ROM), an erasable programmable read-only memory(EPROM), an electrically erasable programmable read-only memory(EEPROM), and a flash memory.

The storage unit 73 stores two or more pieces of 3D data including first3D data and second 3D data. The storage unit 73 may include a firststorage unit storing the first 3D data and a second storage unit storingthe second 3D data.

The first 3D data and the second 3D data include three-dimensionalcoordinates (3D coordinates) of two or more points of a subject. Thefirst 3D data and the second 3D data indicate a 3D shape of the subject.The first 3D data are generated by using two or more first imagesacquired at two or more different viewpoints. The second 3D data aregenerated by using two or more second images acquired at two or moredifferent viewpoints. The two or more first images are not completelythe same as the two or more second images. At least one of the two ormore first images is different from at least one of the two or moresecond images. All the two or more first images may be different fromany of the two or more second images. All the two or more second imagesmay be different from any of the two or more first images.

The first 3D data and the second 3D data are different from each other.At least some of the 3D coordinates included in the first 3D data aredifferent from those included in the second 3D data. One or more pointsincluded in the first 3D data are different from one or more pointsincluded in the second 3D data. At least some of the 3D coordinatesincluded in the second 3D data are different from those included in thefirst 3D data. One or more points included in the second 3D data aredifferent from one or more points included in the first 3D data.

All the 3D coordinates included in the first 3D data may be differentfrom any of the 3D coordinates included in the second 3D data. All thetwo or more points included in the first 3D data may be different fromany of all the two or more points included in the second 3D data. Allthe 3D coordinates included in the second 3D data may be different fromany of the 3D coordinates included in the first 3D data. All the two ormore points included in the second 3D data may be different from any ofall the two or more points included in the first 3D data.

In other words, there is no subject region corresponding to at leastpart of the first 3D data in a subject region corresponding to theentire second 3D data. In addition, there is no subject regioncorresponding to at least part of the second 3D data in a subject regioncorresponding to the entire first 3D data.

A three-dimensional coordinate system (first 3D coordinate system) ofthe first 3D data and a three-dimensional coordinate system (second 3Dcoordinate system) of the second 3D data are different from each other.One of the first 3D coordinate system and the second 3D coordinatesystem can be converted into the other by using a parameter indicatingeach of a position, a posture, and a scale. At least one of theposition, the posture, and the scale is different between the first 3Dcoordinate system and the second 3D coordinate system. The posture isdefined by using three parameters. At least one parameter indicating theposture may be different between the first 3D coordinate system and thesecond 3D coordinate system.

At least some of the two or more viewpoints at which the two or morefirst images are acquired are different from any of the two or moreviewpoints at which the two or more second images are acquired. All thetwo or more viewpoints at which the two or more first images areacquired may be different from any of the two or more viewpoints atwhich the two or more second images are acquired. At least some of thetwo or more viewpoints at which the two or more second images areacquired are different from any of the two or more viewpoints at whichthe two or more first images are acquired. All the two or moreviewpoints at which the two or more second images are acquired may bedifferent from any of the two or more viewpoints at which the two ormore first images are acquired.

The control unit 51 controls each unit of the image display device 50.

The data acquisition unit 52 connects to the storage unit 73 andacquires the first 3D data and the second 3D data from the storage unit73.

The parameter generation unit 53 generates a conversion parameter usedfor converting the first 3D coordinate system and the second 3Dcoordinate system into a common coordinate system. The common coordinatesystem is a 3D coordinate system that is common between the first 3Ddata and the second 3D data. The first 3D data and the second 3D dataare converted into 3D data in the common coordinate system by convertingthe first 3D coordinate system and the second 3D coordinate system intothe common coordinate system. The conversion parameter includes both afirst conversion parameter used for converting the first 3D coordinatesystem into the common coordinate system and a second conversionparameter used for converting the second 3D coordinate system into thecommon coordinate system. Each of the first conversion parameter and thesecond conversion parameter includes both a position-and-postureconversion parameter used for conversion of the position and the postureof each 3D coordinate system and a scale conversion parameter used forconversion of the scale of each 3D coordinate system.

The conversion unit 54 converts the first 3D coordinate system and thesecond 3D coordinate system into the common coordinate system by usingthe conversion parameter generated by the parameter generation unit 53.In this way, the conversion unit 54 converts the first 3D data and thesecond 3D data into 3D data in the common coordinate system.

The data generation unit 55 connects together the first 3D data and thesecond 3D data converted by the conversion unit 54 into the 3D data inthe common coordinate system. In this way, the data generation unit 55generates 3D data of a wide range of a subject.

The display control unit 56 outputs an image of the 3D data to thedisplay unit 71 and displays the image on the display unit 71. Thedisplay control unit 56 displays an image of the first 3D data and thesecond 3D data on the display unit 71. In other words, the displaycontrol unit 56 displays an image of each of the 3D shape indicated bythe first 3D data and the 3D shape indicated by the second 3D data onthe display unit 71. In addition, the display control unit 56 displaysan image of the 3D data generated by the data generation unit 55 on thedisplay unit 71. In other words, the display control unit 56 displays animage of the 3D shape indicated by the 3D data of a wide range of asubject on the display unit 71.

The information acceptance unit 57 accepts information output from theoperation unit 70. Alternatively, the information acceptance unit 57accepts information received by the communication unit 72. Theinformation acceptance unit 57 may accept information corresponding tovoice input into a microphone not shown in FIG. 1 . The informationacceptance unit 57 may accept structure information related to ageometric structure of a subject. The structure information indicates astructure of the subject in a region in which the 3D shape of the first3D data and the 3D shape of the second 3D data are connected together.Hereinafter, the region is called a connection region. The structureinformation is generated without using the first 3D data or the second3D data. The information acceptance unit 57 may accept information notincluding the structure information.

In a case in which the information acceptance unit 57 acceptsinformation not including structure information, the structureestimation unit 58 estimates a structure of a subject in a connectionregion on the basis of the accepted information and generates structureinformation. In a case in which the information acceptance unit 57accepts information including structure information, the structureestimation unit 58 is not used.

Each unit of the image display device 50 may be constituted by at leastone of a processor and a logic circuit. For example, the processor is atleast one of a central processing unit (CPU), a digital signal processor(DSP), and a graphics-processing unit (GPU). For example, the logiccircuit is at least one of an application-specific integrated circuit(ASIC) and a field-programmable gate array (FPGA). Each unit of theimage display device 50 may include one or a plurality of processors.Each unit of the image display device 50 may include one or a pluralityof logic circuits.

A computer of the image display device 50 may read a program and executethe read program. The program includes commands defining the operationsof each unit of the image display device 50. In other words, thefunctions of each unit of the image display device 50 may be realized bysoftware.

The program described above, for example, may be provided by using a“computer-readable storage medium” such as a flash memory. The programmay be transmitted from the computer storing the program to the imagedisplay device 50 through a transmission medium or transmission waves ina transmission medium. The “transmission medium” transmitting theprogram is a medium having a function of transmitting information. Themedium having the function of transmitting information includes anetwork (communication network) such as the Internet and a communicationcircuit line (communication line) such as a telephone line. The programdescribed above may realize some of the functions described above. Inaddition, the program described above may be a differential file(differential program). The functions described above may be realized bya combination of a program that has already been recorded in a computerand a differential program.

Hereinafter, distinctive processing of the first embodiment will bedescribed. In the following descriptions, it is assumed that 3D data aregenerated on the basis of a still image group acquired by endoscopeequipment. The endoscope equipment generates the two or more firstimages used for generating the first 3D data and generates the two ormore second images used for generating the second 3D data. In addition,the two or more first images and the two or more second images aregenerated on the basis of an optical image of a subject acquired througha single-eye optical system. Inspection equipment that acquires a stillimage group is not limited to the endoscope equipment. As long as theinspection equipment includes a camera, the inspection equipment may beany equipment.

For example, each of the two or more first images is a still image, andeach of the two or more second images is a still image. The two or morefirst images may be all or some of two or more images included in avideo. The two or more second images may be all or some of two or moreimages included in a video. The two or more first images may be includedin a first video file, and the two or more second images may be includedin a second video file different from the first video file. The two ormore first images and the two or more second images do not need to bedivided into two video files. The two or more first images and the twoor more second images may be included in a single video file.

Processing executed by the image display device 50 will be described byusing FIG. 2 . FIG. 2 shows a procedure of the processing executed bythe image display device 50.

The data acquisition unit 52 connects to the storage unit 73 andacquires the first 3D data from the storage unit 73 (Step S100). AfterStep S100, the display control unit 56 displays an image of the first 3Ddata on the display unit 71 (Step S101).

After Step S101, the data acquisition unit 52 connects to the storageunit 73 and acquires the second 3D data from the storage unit 73 (StepS102). After Step S102, the display control unit 56 displays an image ofthe second 3D data on the display unit 71 (Step S103).

The order of Steps S100 to S103 is not limited to that shown in FIG. 2 .For example, Step S100 and Step S101 may be executed after Step S102 andStep S103 are executed. Alternatively, Step S101 and Step S103 may beexecuted after Step S100 and Step S102 are executed. Step S101 and StepS103 may be omitted.

FIG. 3 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG11 on the display unit71. The image IMG11 includes a region R11 and a region R12. An image ofa first 3D shape 3D11 is displayed in the region R11. The first 3D shape3D11 is indicated by the first 3D data. An image of a second 3D shape3D12 is displayed in the region R12. The second 3D shape 3D12 isindicated by the second 3D data.

After Step S103, the information acceptance unit 57 accepts informationof an inspection target (Step S104). Hereinafter, an example in whichthe information acceptance unit 57 accepts information not includingstructure information from the operation unit 70 will be described.

The display control unit 56 displays a dialog box on the display unit71. FIG. 4 shows an example of the dialog box displayed on the displayunit 71. The display control unit 56 displays a dialog box DL10 on thedisplay unit 71. A user can select information in the dialog box DL10 byoperating the operation unit 70.

A cursor CS10 and a cursor CS11 are displayed in the dialog box DL10.Each of the cursor CS10 and the cursor CS11 indicates an item selectedat present. Information INF10 is displayed in the dialog box DL10. Theinformation INF10 includes a name of an inspection target and includes aname of a typical inspection portion or structure that is present in theinspection target.

A user selects a subject in the inspection target. In a case in which aregion in which the first 3D shape 3D11 and the second 3D shape 3D12shown in FIG. 3 are connected together is a “straight pipe,” a userselects a “pipe” shown in FIG. 4 and then selects a “straight pipe.”When the user inputs information indicating that the selection isfinalized into the image display device 50, the information acceptanceunit 57 accepts the information selected by the user. At this time, theinformation acceptance unit 57 accepts a character string “straightpipe.”

After Step S104, the structure estimation unit 58 estimates a structureof a subject in a connection region on the basis of the informationaccepted in Step S104 and generates structure information (Step S105).

An example of a method in which the structure estimation unit 58generates the structure information will be described. For example, thestructure estimation unit 58 uses a reference table TB11 shown in FIG. 5. The reference table TB11 includes information (input information)accepted by the information acceptance unit 57 and the structureinformation. The input information and the structure information areassociated with each other.

The structure estimation unit 58 converts a character string accepted bythe information acceptance unit 57 into structure information. Forexample, when the information acceptance unit 57 accepts a characterstring “straight pipe,” the structure estimation unit 58 refers to acolumn CL11 in the reference table TB11 and acquires the structureinformation in the column CL11. For example, the structure informationindicates that the 3D shape of the first 3D data and the 3D shape of thesecond 3D data are almost a cylinder. Since there is a possibility thatthe 3D data include a foreign substance or the like, the structureinformation indicates an approximate cylinder rather than a completecylinder. The structure information indicates that the inner diameter ofthe 3D shape of the first 3D data and the inner diameter of the 3D shapeof the second 3D data are the same. The structure information indicatesthat the center axis of the cylinder of the first 3D data matches thecenter axis of the cylinder of the second 3D data.

A method of generating the structure information is not limited to thatusing the reference table TB11. For example, the information acceptanceunit 57 may accept similar information to the structure informationincluded in the reference table TB11 from the operation unit 70. Inother words, the information acceptance unit 57 may accept informationincluding the structure information from the operation unit 70. In thiscase, the structure estimation unit 58 may be omitted.

The display control unit 56 may display an input box on the display unit71. A user may input a word or a keyword into the input box by using theoperation unit 70 or a microphone. The structure estimation unit 58 mayestimate the structure of the subject on the basis of the input word orkeyword and may generate structure information.

A method of acquiring structure information of a connection region isnot limited to that described above. For example, the structureestimation unit 58 may estimate the structure of the subject by using atechnique such as artificial intelligence (AI).

After Step S105, the parameter generation unit 53 generates a conversionparameter used for converting the first 3D coordinate system and thesecond 3D coordinate system into a common coordinate system (Step S106).

Hereinafter, a 3D coordinate system will be described. The first 3Dcoordinate system in the first 3D data and the second 3D coordinatesystem in the second 3D data do not match each other. In order tocombine the first 3D data and the second 3D data into 3D data of a widerange, the first 3D coordinate system and the second 3D coordinatesystem need to be converted into any common coordinate system. Thecommon coordinate system may be a different 3D coordinate system fromany of the first 3D coordinate system and the second 3D coordinatesystem. The common coordinate system may be the same as the first 3Dcoordinate system or the second 3D coordinate system.

In a case in which the common coordinate system is different from any ofthe first 3D coordinate system and the second 3D coordinate system, theparameter generation unit 53 generates a first conversion parameter usedfor converting the first 3D coordinate system into a third 3D coordinatesystem and generates a second conversion parameter used for convertingthe second 3D coordinate system into the third 3D coordinate system.

On the other hand, in a case in which the common coordinate system isthe same as the first 3D coordinate system, the parameter generationunit 53 generates only a second conversion parameter used for convertingthe second 3D coordinate system into the first 3D coordinate system.Similarly, in a case in which the common coordinate system is the sameas the second 3D coordinate system, the parameter generation unit 53generates only a first conversion parameter used for converting thefirst 3D coordinate system into the second 3D coordinate system.

Hereinafter, a scale of a 3D coordinate system will be described. Thescale of the first 3D data and the scale of the second 3D data do notnecessarily match each other. In order to combine the first 3D data andthe second 3D data into 3D data of a wide range, the scale of the first3D data and the scale of the second 3D data need to match each other.The scale of the combined 3D data may be different from any of the scaleof the first 3D data and the scale of the second 3D data. The scale ofthe combined 3D data may be the same as the scale of the first 3D dataor the scale of the second 3D data.

Hereinafter, an example in which the first 3D coordinate system is thereference of coordinate systems and is used as a common coordinatesystem will be described. In the following example, the parametergeneration unit 53 generates a position-and-posture conversion parameterused for causing the position and the posture of the second 3Dcoordinate system to match the position and the posture of the first 3Dcoordinate system, respectively. In addition, the parameter generationunit 53 generates a scale conversion parameter used for causing thescale of the second 3D coordinate system to match the scale of the first3D coordinate system. However, a method of generating a conversionparameter is not limited to the following example. The second 3Dcoordinate system may be used as a common coordinate system. A different3D coordinate system from any of the first 3D coordinate system and thesecond 3D coordinate system may be used as a common coordinate system.

The parameter generation unit 53 executes processing shown in FIG. 6 inStep S106. FIG. 6 shows a procedure of the processing executed by theparameter generation unit 53.

The structure information indicates that the 3D shape of the first 3Ddata and the 3D shape of the second 3D data are almost a cylinder.Therefore, the parameter generation unit 53 calculates a cylinder axis(first cylinder axis) of the first 3D data (Step S106 a). The cylinderaxis of the first 3D data indicates a center axis of a cylinder of thefirst 3D data.

FIG. 7 shows a similar image to that shown in FIG. 3 . The same parts asthose shown in FIG. 3 will not be described. The parameter generationunit 53 calculates a first cylinder axis AX11.

After Step S106 a, the parameter generation unit 53 calculates acylinder axis (second cylinder axis) of the second 3D data (Step S106b). The cylinder axis of the second 3D data indicates a center axis of acylinder of the second 3D data. The parameter generation unit 53calculates a second cylinder axis AX12 shown in FIG. 7 .

The order of Step S106 a and Step S106 b is not limited to that shown inFIG. 6 . For example, Step S106 a may be executed after Step S106 b isexecuted.

The structure information indicates that the inner diameter of the 3Dshape of the first 3D data and the inner diameter of the 3D shape of thesecond 3D data are the same. Therefore, after Step S106 b, the parametergeneration unit 53 generates a scale conversion parameter used forcorrecting the scale of the second 3D data such that the inner diameterof the cylinder of the first 3D data and the inner diameter of thecylinder of the second 3D data match each other (Step S106 c).Specifically, the parameter generation unit 53 generates a scaleconversion parameter used for causing a diameter DM12 of the cylinder ofthe second 3D shape 3D12 shown in FIG. 7 to match a diameter DM11 of thecylinder of the first 3D shape 3D11.

The structure information indicates that the center axis of the cylinderof the first 3D data and the center axis of the cylinder of the second3D data match each other. Therefore, after Step S106 c, the parametergeneration unit 53 generates a position-and-posture conversion parameterused for correcting the position and the posture of the second 3D datasuch that the cylinder axis of the first 3D data and the cylinder axisof the second 3D data match each other (Step S106 d). When Step S106 dis executed, the processing shown in FIG. 6 is completed.

In a case in which each piece of 3D data includes a timestamp (timeinformation), the parameter generation unit 53 can identify a connectionregion by using the timestamp. The timestamp indicates a time point atwhich each of two or more images used for generating each of the first3D data and the second 3D data are generated. For example, the first 3Ddata include timestamps from a time point t1 to a time point t2 shown inFIG. 7 , and the second 3D data include timestamps from a time point t3to a time point t4 shown in FIG. 7 .

The parameter generation unit 53 identifies a region of the first 3Ddata associated with the time point t2 and a region of the second 3Ddata associated with the time point t3 as connection regions.Specifically, the parameter generation unit 53 identifies a firstconnection region CR11 of the first 3D shape 3D11 and a secondconnection region CR12 of the second 3D shape 3D12.

The first connection region CR11 includes the terminal end of the first3D shape 3D11. The second connection region CR12 includes the start endof the second 3D shape 3D12. The parameter generation unit 53 generatesa position-and-posture conversion parameter used for causing the secondcylinder axis AX12 in the second connection region CR12 to match thefirst cylinder axis AX11 in the first connection region CR11.

The parameter generation unit 53 can convert the first 3D coordinatesystem and the second 3D coordinate system into a common coordinatesystem by executing Steps S106 a to S106 d. When the informationacceptance unit 57 accepts a character string “straight pipe,” theparameter generation unit 53 generates a position-and-posture conversionparameter and a scale conversion parameter used for converting the first3D coordinate system and the second 3D coordinate system into a commoncoordinate system.

After Step S106, the conversion unit 54 converts the first 3D coordinatesystem and the second 3D coordinate system into a common coordinatesystem by using the conversion parameter generated in Step S106. Inother words, the conversion unit 54 converts the first 3D data and thesecond 3D data into 3D data in the common coordinate system (Step S107).The position, the posture, and the scale of the first 3D coordinatesystem are not changed, and the position, the posture, and the scale ofthe second 3D coordinate system are changed in a case in which the first3D coordinate system is used as the common coordinate system.

After Step S107, the data generation unit 55 connects the first 3D dataand the second 3D data together in the common coordinate system (StepS108). In this way, the data generation unit 55 generates 3D data of awide range of a subject.

After the first 3D coordinate system and the second 3D coordinate systemare converted into the common coordinate system, the first 3D data andthe second 3D data do not need to be combined into a single piece of 3Ddata. Therefore, the data generation unit 55 may be omitted.

After Step S108, the display control unit 56 displays an image of a 3Dshape indicated by the 3D data generated in Step S108 on the displayunit 71 (Step S109). When Step S109 is executed, the processing shown inFIG. 2 is completed.

FIG. 8 shows an example of an image displayed on the display unit 71.The same parts as those shown in FIG. 7 will not be described. Thedisplay control unit 56 displays an image IMG12 on the display unit 71.The image IMG12 includes a region R13. An image of a first 3D shape 3D11and a second 3D shape 3D12 is displayed in the region R13.

The first 3D shape 3D11 and the second 3D shape 3D12 are disposed suchthat a first connection region and a second connection region areconnected together. In the example shown in FIG. 8 , a gap is shownbetween the first 3D shape 3D11 and the second 3D shape 3D12 in order tofacilitate understanding of connection between the first connectionregion and the second connection region.

The display control unit 56 displays a region R14 and a region R15 inthe region R13. The region R14 corresponds to the first connectionregion, and the region R15 corresponds to the second connection region.In the example shown in FIG. 8 , each of the region R14 and the regionR15 is shown by a line. Since each of the region R14 and the region R15is shown by a line, a user can easily check the position at which thefirst 3D shape 3D11 and the second 3D shape 3D12 are connected together.

The display control unit 56 may display the region R14 and the regionR15 in a first color and may display a region other than the region R14or the region R15 in a second color different from the first color. Theposition of each of the region R14 and the region R15 may be reported toa user by using voice. As long as a user can distinguish a connectionregion from the other regions, a method of reporting a region of a 3Dshape to the user is not limited to the above-described method.

The display control unit 56 may display the region R14 and the regionR15 in different colors. When a user looks over a 3D shape of a widerange, the user can easily check how far apart the first 3D shape 3D11and the second 3D shape 3D12 are from each other. In addition, the usercan easily check where the first 3D shape 3D11 and the second 3D shape3D12 are connected together.

An image of the first 3D shape 3D11 and the second 3D shape 3D12 may beattached to an inspection report. The display control unit 56 maydisplay the inspection report on the display unit 71, thus displayingthe image of the first 3D shape 3D11 and the second 3D shape 3D12 on thedisplay unit 71.

The parameter generation unit 53 may calculate the reliability ofconnection between the first connection region and the second connectionregion in Step S106. The reliability indicates the accuracy of theconnection between the first connection region and the second connectionregion. For example, the parameter generation unit 53 executes cylinderfitting by using the first 3D data. In this way, the parametergeneration unit 53 calculates a cylinder approximating the first 3D dataand calculates a center axis (first cylinder axis) of the cylinder. Inaddition, the parameter generation unit 53 executes cylinder fitting byusing the second 3D data. In this way, the parameter generation unit 53calculates a cylinder approximating the second 3D data and calculates acenter axis (second cylinder axis) of the cylinder.

An error occurs between the 3D coordinates included in the 3D data andthe 3D coordinates of the calculated cylinder. The parameter generationunit 53 calculates an error of the distance between each point of the 3Ddata and each point on the cylinder for all the points of the 3D dataand calculates an average, a standard deviation, or the like of theerror as the reliability of the 3D data. The parameter generation unit53 calculates a statistic of the reliability of the first 3D data and astatistic of the reliability of the second 3D data as the reliability ofthe connection between the first connection region and the secondconnection region. The statistic is a minimum value, a maximum value, anaverage value, or the like of the reliability of the first 3D data andthe reliability of the second 3D data.

The display control unit 56 may display the reliability of the first 3Ddata and the reliability of the second 3D data on the display unit 71 inStep S109. For example, the reliability is displayed as percentage.Alternatively, the reliability is displayed on a scale of 1 to 5. Amethod of displaying the reliability is not limited to theabove-described examples.

In a case in which the 3D shape indicated by the 3D data is a completecylinder, the parameter generation unit 53 can accurately calculate acylinder axis. Therefore, the reliability is high. On the other hand, ina case in which the 3D data include an error, the center axis of acylinder approximating the 3D data is shifted from an original centeraxis. Therefore, the reliability is low. Since the reliability isdisplayed, a user can check the accuracy of the connection between thefirst connection region and the second connection region.

A three-dimensional image display method according to each aspect of thepresent invention includes a first acquisition step, a secondacquisition step, a conversion step, and a display step. The dataacquisition unit 52 connects to the storage unit 73 (recording medium)storing first 3D data of a subject and second 3D data of the subject andacquires the first 3D data from the storage unit 73 in the firstacquisition step (Step S100). The first 3D data include 3D coordinatesdefined in a first 3D coordinate system. The second 3D data include 3Dcoordinates defined in a second 3D coordinate system different from thefirst 3D coordinate system. At least part of a region of the subjectcorresponding to the first 3D data is different from at least part of aregion of the subject corresponding to the second 3D data. The dataacquisition unit 52 connects to the storage unit 73 and acquires thesecond 3D data from the storage unit 73 in the second acquisition step(Step S102). The conversion unit 54 converts the first 3D coordinatesystem and the second 3D coordinate system into a three-dimensionalcommon coordinate system on the basis of structure information relatedto a geometric structure of the subject in the conversion step (StepS107). The structure information is generated without using the first 3Ddata or the second 3D data. The display control unit 56 displays animage of the first 3D data in the common coordinate system and an imageof the second 3D data in the common coordinate system on the displayunit 71 (display) in the display step (Step S109).

A three-dimensional image display device according to each aspect of thepresent invention includes the data acquisition unit 52, the conversionunit 54, and the display control unit 56. The data acquisition unit 52connects to the storage unit 73 (recording medium) storing first 3D dataof a subject and second 3D data of the subject and acquires the first 3Ddata and the second 3D data from the storage unit 73. The conversionunit 54 converts a first 3D coordinate system and a second 3D coordinatesystem into a three-dimensional common coordinate system on the basis ofstructure information related to a geometric structure of the subject.The display control unit 56 displays an image of the first 3D data inthe common coordinate system and an image of the second 3D data in thecommon coordinate system on the display unit 71 (display).

Each aspect of the present invention may include the following modifiedexample. Each of two or more first images and each of two or more secondimages include time information (timestamp). The conversion unit 54converts the first 3D coordinate system and the second 3D coordinatesystem into the common coordinate system on the basis of the timeinformation in the conversion step (Step S107).

Each aspect of the present invention may include the following modifiedexample. The conversion unit 54 converts the first 3D coordinate systemand the second 3D coordinate system into the common coordinate systemsuch that a first region (first connection region CR11) of the subjectand a second region (second connection region CR12) of the subject areconnected together in the conversion step (Step S107). The first regioncorresponds to the 3D coordinates included in the first 3D data. Thesecond region corresponds to the 3D coordinates included in the second3D data. The display control unit 56 displays information indicating thepositions of the first region and the second region on the display unit71 (display) in the display step (Step S109).

Each aspect of the present invention may include the following modifiedexample. The display control unit 56 displays information indicating theaccuracy of connection between the first region (first connection regionCR11) and the second region (second connection region CR12) on thedisplay unit 71 (display) in the display step (Step S109).

Each aspect of the present invention may include the following modifiedexample. The conversion unit 54 generates a position conversionparameter and a posture conversion parameter used for converting thefirst 3D coordinate system and the second 3D coordinate system into thecommon coordinate system on the basis of the structure information inthe conversion step (Step S107). The conversion unit 54 converts thefirst 3D coordinate system and the second 3D coordinate system into thecommon coordinate system by using the position conversion parameter andthe posture conversion parameter in the conversion step (Step S107).

Each aspect of the present invention may include the following modifiedexample. The parameter generation unit 53 generates a scale conversionparameter used for correcting at least one of a scale of a 3D shapeindicated by the first 3D data and a scale of a 3D shape indicated bythe second 3D data in a generation step (Step S106). The conversion unit54 converts the first 3D coordinate system and the second 3D coordinatesystem into the common coordinate system by using the scale conversionparameter in the conversion step (Step S107).

Each aspect of the present invention may include the following modifiedexample. The first 3D data are generated by using two or more firstimages acquired at two or more different viewpoints. The second 3D dataare generated by using two or more second images acquired at two or moredifferent viewpoints. At least one of the two or more first images andat least one of the two or more second images are different from eachother.

Each aspect of the present invention may include the following modifiedexample. The two or more first images are at least some of two or moreimages included in a first video. The two or more second images are atleast some of two or more images included in a second video that is thesame as or different from the first video.

Each aspect of the present invention may include the following modifiedexample. The two or more first images and the two or more second imagesare generated by an endoscope.

Each aspect of the present invention may include the following modifiedexample. The two or more first images and the two or more second imagesare generated on the basis of an optical image of the subject acquiredthrough a single-eye optical system.

In the first embodiment, there is a case in which the first 3D data andthe second 3D data do not include 3D data of a common region. Even insuch a case, the image display device 50 can display an image of a 3Dshape of a wide range indicated by two or more pieces of 3D data.Therefore, a user can capture the entire structure of an inspectiontarget in a visual field and can check the structure.

There is a case in which 3D data corresponding to a region between afirst connection region in the 3D shape indicated by the first 3D dataand a second connection region in the 3D shape indicated by the second3D data are lost. Even in such a case, the image display device 50 candisplay an image of a 3D shape of a wide range indicated by two or morepieces of 3D data.

First Modified Example of First Embodiment

A first modified example of the first embodiment of the presentinvention will be described. The image display device 50 shown in FIG. 1is changed to an image display device 50 a shown in FIG. 9 . FIG. 9shows a configuration of the image display device 50 a. The sameconfiguration as that shown in FIG. 1 will not be described.

The image display device 50 a shown in FIG. 9 includes a control unit51, a data acquisition unit 52, a parameter generation unit 53, aconversion unit 54, a data generation unit 55, a display control unit56, an information acceptance unit 57, a structure estimation unit 58,and an adjustment unit 59.

Each unit of the image display device 50 a may be constituted by atleast one of a processor and a logic circuit. Each unit of the imagedisplay device 50 a may include one or a plurality of processors. Eachunit of the image display device 50 a may include one or a plurality oflogic circuits.

The adjustment unit 59 adjusts at least one of the position and theposture of at least one of the first 3D data and the second 3D data inthe common coordinate system. For example, the adjustment unit 59executes adjustment processing of the first 3D data and the second 3Ddata. Alternatively, the adjustment unit 59 executes adjustmentprocessing of only the first 3D data or adjustment processing of onlythe second 3D data. The adjustment unit 59 adjusts the position and theposture of 3D data in the adjustment processing. Alternatively, theadjustment unit 59 adjusts only the position of the 3D data or only theposture of the 3D data in the adjustment processing.

After the adjustment unit 59 executes the adjustment processing, thedisplay control unit 56 displays an image of the first 3D data and animage of the second 3D data on the display unit 71 again. In this way,at least one of the position and the posture of at least one of theimage of the first 3D data and the image of the second 3D data ischanged.

Processing executed by the image display device 50 a will be describedby using FIG. 10 . FIG. 10 shows a procedure of the processing executedby the image display device 50 a. The same processing as that shown inFIG. 2 will not be described.

For example, the image IMG12 shown in FIG. 8 is displayed on the displayunit 71 in Step S108. In the example shown in FIG. 8 , the terminal endof the first 3D shape 3D11 and the start end of the second 3D shape 3D12match each other. There is a case in which 3D data between the terminalend of the first 3D shape 3D11 and the start end of the second 3D shape3D12 are lost.

In the first modified example of the first embodiment, a user estimatesthe length of the section between the terminal end of the first 3D shape3D11 and the start end of the second 3D shape 3D12. The 3D data in thesection are lost. A user can change the disposition of the first 3Dshape 3D11 and the second 3D shape 3D12 such that the terminal end ofthe first 3D shape 3D11 and the start end of the second 3D shape 3D12are away from each other by the length of the section.

A user inputs an instruction to move at least one of the first 3D shape3D11 and the second 3D shape 3D12 into the image display device 50 a byoperating the operation unit 70. The information acceptance unit 57accepts the instruction. Hereinafter, an example in which the second 3Dshape 3D12 moves will be described. The adjustment unit 59 adjusts atleast one of the position and the posture of the second 3D shape 3D12 onthe basis of the instruction accepted by the information acceptance unit57 (Step S110).

The 3D data generated in Step S108 include 3D data corresponding to thefirst 3D data and 3D data corresponding to the second 3D data. Theadjustment unit 59 changes the 3D coordinates of the 3D datacorresponding to the second 3D data. In this way, the adjustment unit 59adjusts at least one of the position and the posture of the 3D shapeindicated by the 3D data.

After Step S110, the display control unit 56 displays an image of the 3Dshape indicated by the 3D data processed in Step S110 on the displayunit 71 (Step S110. When Step S111 is executed, the processing shown inFIG. 10 is completed.

FIG. 11 shows an example of an image displayed on the display unit 71.The same parts as those shown in FIG. 8 will not be described. Thedisplay control unit 56 displays an image IMG13 on the display unit 71.

A user inputs an instruction to bring the second 3D shape 3D12 closer tothe first 3D shape 3D11 or an instruction to put the second 3D shape3D12 away from the first 3D shape 3D11 into the image display device 50a. Alternatively, the user inputs an instruction to rotate the second 3Dshape 3D12 into the image display device 50 a.

An operation performed by a user in order to freely change the positionand the posture of the second 3D shape 3D12 is complicated. Therefore,movement of the second 3D shape 3D12 may be restricted. For example,when a “straight pipe” is selected, the user may move the second 3Dshape 3D12 in a range in which the direction of the cylinder axis doesnot change. In this case, the user can move the second 3D shape 3D12 ina parallel direction to the cylinder axis and can rotate the second 3Dshape 3D12 around the cylinder axis.

Even when a subject is a straight pipe, a condition for restrictingmovement of the second 3D shape 3D12 is not limited to theabove-described example.

In the example shown in FIG. 11 , a user inputs an instruction to movethe second 3D shape 3D12 in the right direction into the image displaydevice 50 a. The adjustment unit 59 changes the 3D coordinates of the 3Ddata such that the second 3D shape 3D12 moves in the right direction.

Each aspect of the present invention may include the following modifiedexample. The adjustment unit 59 adjusts at least one of the position andthe posture of at least one of the first 3D data and the second 3D datain the common coordinate system in an adjustment step (Step S110).

In the first modified example of the first embodiment, at least one ofthe position and the posture of the first 3D shape and the second 3Dshape in the 3D shape of the wide range is adjusted. Since the 3D shapeindicated by the 3D data of the wide range approximates the structure ofan actual inspection target as a result of this adjustment, the qualityof the 3D data is improved.

Second Modified Example of First Embodiment

A second modified example of the first embodiment of the presentinvention will be described. In the second modified example of the firstembodiment, the image display device 50 shown in FIG. 1 is used.

A subject in the first embodiment described above is a straight pipe. Onthe other hand, in the second modified example of the first embodiment,an example in which the subject is a 90-degree fitting will bedescribed.

For example, a user selects a “pipe” shown in FIG. 4 and then selects a“90-degree fitting.” When the user inputs information indicating thatthe selection is finalized into the image display device 50, theinformation acceptance unit 57 accepts the information selected by theuser. At this time, the information acceptance unit 57 accepts acharacter string “90-degree fitting.”

For example, structure information of a 90-degree fitting indicates thatthe 3D shape of the first 3D data and the 3D shape of the second 3D dataare almost a cylinder. The structure information indicates that theinner diameter of the 3D shape of the first 3D data and the innerdiameter of the 3D shape of the second 3D data are the same. Thestructure information indicates that the center axis of the cylinder ofthe first 3D data is orthogonal to the center axis of the cylinder ofthe second 3D data. The parameter generation unit 53 generates aconversion parameter on the basis of the structure information in StepS106.

FIG. 12 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG21 on the display unit71. The image IMG21 includes a region R21. An image of a first 3D shape3D21 and a second 3D shape 3D22 is displayed in the region R21. Thefirst 3D shape 3D21 is indicated by the first 3D data. The second 3Dshape 3D22 is indicated by the second 3D data.

In the example shown in FIG. 12 , the parameter generation unit 53generates a position-and-posture conversion parameter used for causingthe position and the posture of the second 3D coordinate system to matchthe position and the posture of the first 3D coordinate system,respectively. In addition, the parameter generation unit 53 generates ascale conversion parameter used for causing the scale of the second 3Dcoordinate system to match the scale of the first 3D coordinate system.The parameter generation unit 53 calculates a first cylinder axis AX21of the first 3D shape 3D21 and a second cylinder axis AX22 of the second3D shape 3D22. The parameter generation unit 53 generates a scaleconversion parameter used for causing a diameter DM22 of the cylinder ofthe second 3D shape 3D22 to match a diameter DM21 of the cylinder of thefirst 3D shape 3D21.

The first 3D data include timestamps from a time point t1 to a timepoint t2 shown in FIG. 12 , and the second 3D data include timestampsfrom a time point t3 to a time point t4 shown in FIG. 12 . The parametergeneration unit 53 identifies a first connection region of the first 3Dshape 3D21 associated with the time point t2 and identifies a secondconnection region of the second 3D shape 3D22 associated with the timepoint t3.

The first connection region includes the terminal end of the first 3Dshape 3D21. The second connection region includes the start end of thesecond 3D shape 3D22. The parameter generation unit 53 generates aposition-and-posture conversion parameter used for causing the secondcylinder axis AX22 in the second connection region to be orthogonal tothe first cylinder axis AX21 in the first connection region.

The second connection region is disposed at a position close to thefirst connection region. The parameter generation unit 53 generates theposition-and-posture conversion parameter causing a region between thefirst connection region and the second connection region to be small.The 3D data of the region (a region of a 90-degree fitting) between thefirst connection region and the second connection region are lost. Theparameter generation unit 53 generates the position-and-postureconversion parameter such that the first connection region and thesecond connection region do not overlap each other.

There is a possibility that 3D data of a straight pipe part are lost inaddition to 3D data of a curved region. Therefore, the length of asection between the first connection region and the second connectionregion is actually unknown. After the image IMG21 shown in FIG. 12 isdisplayed, the position or the posture of the 3D shape may be adjustedas in the first modified example of the first embodiment.

The display control unit 56 displays a region R22 and a region R23 inthe region R21. The region R22 corresponds to the first connectionregion, and the region R23 corresponds to the second connection region.

The purpose of each embodiment of the present invention is to display animage of a 3D shape of a wide range even when the first 3D data and thesecond 3D data do not include 3D data of a common region. Therefore, thepositions and the postures of the first 3D data and the second 3D datado not need to be accurately adjusted. A user may adjust the positionsand the postures. The first 3D data and the second 3D data may beroughly connected together.

FIG. 13 shows an example of an image displayed on the display unit 71.The same parts as those shown in FIG. 12 will not be described. Thedisplay control unit 56 displays an image IMG22 on the display unit 71.

A point on a first cylinder axis AX21 at the terminal end of a first 3Dshape 3D21 matches a point on a second cylinder axis AX22 at the startend of a second 3D shape 3D22. Part of a first connection region and apart of a second connection region overlap each other. The structure ofa subject, which is an inspection target, is actually different fromthat shown in FIG. 13 . In a case in which the first 3D data and thesecond 3D data do not need to be accurately connected together, thedisplay control unit 56 can display the image IMG22 on the display unit71, as shown in FIG. 13 . A user may adjust the position and the postureof the 3D data as in the first modified example of the first embodimentin order to improve the appearance of a 3D shape of a wide range.

In the second modified example of the first embodiment, in a case inwhich a subject is a 90-degree fitting, the image display device 50executes similar processing to that of selecting a straight pipe as asubject. In this way, the image display device 50 can connect the first3D data and the second 3D data together and can display an image of a 3Dshape of a wide range.

(Third modified example of first embodiment) A third modified example ofthe first embodiment of the present invention will be described. In thethird modified example of the first embodiment, the image display device50 shown in FIG. 1 is used.

A subject in the second modified example of the first embodimentdescribed above is a 90-degree fitting. On the other hand, in the thirdmodified example of the first embodiment, an example in which a subjectis a 90-degree fitting having different diameters will be described.There is a case in which a fitting having different diameters is usedfor a pipe. The third modified example of the first embodiment isapplied to an inspection of such a pipe.

After the dialog box DL10 shown in FIG. 4 is displayed on the displayunit 71, the display control unit 56 displays a dialog box DL20 shown inFIG. 14 on the display unit 71. A user inputs the inner diameter of thefirst 3D data into an input box IB21 and inputs the inner diameter ofthe second 3D data into an input box IB22 by operating the operationunit 70.

The information acceptance unit 57 accepts the inner diameter input intothe input box IB21 and the inner diameter input into the input box IB22.The parameter generation unit 53 generates a scale conversion parameteron the basis of the inner diameter of the first 3D data and the innerdiameter of the second 3D data in Step S106.

An example in which a subject is a 90-degree fitting and the innerdiameter of the cylinder of the second 3D data is 1.5 times as large asthat of the cylinder of the first 3D data will be described. FIG. 15shows an example of an image displayed on the display unit 71. Thedisplay control unit 56 displays an image IMG31 on the display unit 71.The image IMG31 includes a region R31. An image of a first 3D shape 3D31and a second 3D shape 3D32 is displayed in the region R31. The first 3Dshape 3D31 is indicated by the first 3D data. The second 3D shape 3D32is indicated by the second 3D data.

In the example shown in FIG. 15 , the parameter generation unit 53generates a position-and-posture conversion parameter used for causingthe position and the posture of the second 3D coordinate system to matchthe position and the posture of the first 3D coordinate system,respectively. In addition, the parameter generation unit 53 generates ascale conversion parameter used for causing the scale of the second 3Dcoordinate system to match the scale of the first 3D coordinate system.The parameter generation unit 53 calculates a first cylinder axis AX31of the first 3D shape 3D31 and a second cylinder axis AX32 of the second3D shape 3D32. The parameter generation unit 53 generates a scaleconversion parameter used for causing a diameter DM32 of the cylinder ofthe second 3D shape 3D32 to be 1.5 times as large as a diameter DM31 ofthe cylinder of the first 3D shape 3D31.

The first 3D data include timestamps from a time point t1 to a timepoint t2 shown in FIG. 15 , and the second 3D data include timestampsfrom a time point t3 to a time point t4 shown in FIG. 15 . The parametergeneration unit 53 identifies a first connection region of the first 3Dshape 3D31 associated with the time point t2 and identifies a secondconnection region of the second 3D shape 3D32 associated with the timepoint t3.

The first connection region includes the terminal end of the first 3Dshape 3D31. The second connection region includes the start end of thesecond 3D shape 3D32. The parameter generation unit 53 generates aposition-and-posture conversion parameter used for causing the secondcylinder axis AX32 in the second connection region to be orthogonal tothe first cylinder axis AX31 in the first connection region.

The second connection region is disposed at a position close to thefirst connection region. The parameter generation unit 53 generates theposition-and-posture conversion parameter causing a region between thefirst connection region and the second connection region to be small.The 3D data of the region between the first connection region and thesecond connection region are lost. The parameter generation unit 53generates the position-and-posture conversion parameter such that thefirst connection region and the second connection region do not overlapeach other. After an image of 3D data of a wide range is displayed, auser may adjust the position and the posture of the 3D data as in thefirst modified example of the first embodiment.

The display control unit 56 displays a region R32 and a region R33 inthe region R31. The region R32 corresponds to the first connectionregion, and the region R33 corresponds to the second connection region.

In the third modified example of the first embodiment, in a case inwhich a subject is a 90-degree fitting having different diameters, theimage display device 50 executes similar processing to that of selectinga straight pipe as a subject. In this way, the image display device 50can connect the first 3D data and the second 3D data together and candisplay an image of a 3D shape of a wide range.

Fourth Modified Example of First Embodiment

A fourth modified example of the first embodiment of the presentinvention will be described. In the fourth modified example of the firstembodiment, the image display device 50 shown in FIG. 1 is used.

A subject in the first embodiment described above is a straight pipe. Asubject in the second and third modified examples of the firstembodiment described above is a 90-degree fitting. On the other hand, inthe fourth modified example of the first embodiment, an example in whicha subject is a u-shaped tube will be described. A heat exchange tube orthe like includes a u-shaped tube.

FIG. 16 shows an example of a u-shaped tube. The u-shaped tube includesa straight pipe part SP41 and a circular arc part CP41. Hereinafter, anexample in which part of the 3D data are lost in the circular arc partCP41 will be described. The first 3D data and the second 3D data areconnected together in the circular arc part CP41. In a case in which thefirst 3D data and the second 3D data are connected together in thestraight pipe part SP41, the image display device 50 executes similarprocessing to that in the first embodiment in which a subject is astraight pipe, and connects the first 3D data and the second 3D datatogether.

For example, structure information of a u-shaped tube indicates that the3D shape of the first 3D data and the 3D shape of the second 3D data arealmost a cylinder. The structure information indicates that the circulararc part is almost donut-shaped. The range of the circular arc part ofwhich 3D data exist is only a range of 180 degrees, that is, asemicircle. The structure information indicates that the inner diameterof the 3D shape of the first 3D data and the inner diameter of the 3Dshape of the second 3D data are the same. The structure informationindicates that the two center axes of the two cylinders in the straightpipe part are parallel to each other and the circular arc part has anarbitrary curvature radius. The parameter generation unit 53 generates aconversion parameter on the basis of the structure information in StepS106.

FIG. 17 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG41 on the display unit71. The image IMG41 includes a region R41. An image of a first 3D shape3D41 and a second 3D shape 3D42 is displayed in the region R41. Thefirst 3D shape 3D41 is indicated by the first 3D data. The second 3Dshape 3D42 is indicated by the second 3D data.

In the example shown in FIG. 17 , the parameter generation unit 53generates a position-and-posture conversion parameter used for causingthe position and the posture of the second 3D coordinate system to matchthe position and the posture of the first 3D coordinate system,respectively. In addition, the parameter generation unit 53 generates ascale conversion parameter used for causing the scale of the second 3Dcoordinate system to match the scale of the first 3D coordinate system.The parameter generation unit 53 calculates a first cylinder axis AX41of the first 3D shape 3D41 and a second cylinder axis AX42 of the second3D shape 3D42. Each of the first cylinder axis AX41 and the secondcylinder axis AX42 has a straight line shape in the straight pipe partand has a circular arc shape in the circular arc part.

The parameter generation unit 53 generates a scale conversion parameterused for causing a diameter DM42 of the cylinder of the second 3D shape3D42 to match a diameter DM41 of the cylinder of the first 3D shape3D41. In many cases, two inner diameters of two cylinders in a straightpipe part of a u-shaped tube are the same. Therefore, the image displaydevice 50 can generate the scale conversion parameter by using similarmethod to that of generating a scale conversion parameter in the firstembodiment.

The first 3D data include timestamps from a time point t1 to a timepoint t2 shown in FIG. 17 , and the second 3D data include timestampsfrom a time point t3 to a time point t4 shown in FIG. 17 . The parametergeneration unit 53 identifies a first connection region of the first 3Dshape 3D41 associated with the time point t2 and identifies a secondconnection region of the second 3D shape 3D42 associated with the timepoint t3.

The first connection region includes the terminal end of the first 3Dshape 3D41. The second connection region includes the start end of thesecond 3D shape 3D42. The parameter generation unit 53 generates aposition-and-posture conversion parameter used for connecting the firstconnection region and the second connection region together. The firstconnection region and the second connection region are connectedtogether on condition that the circular arc part has a predeterminedcurvature radius.

In a case in which part of the 3D data in the circular arc part is lost,information indicating the angle of the circular arc part correspondingto the position of the connection region is not obtained. In a case inwhich the second 3D data include the 3D data of the straight pipe part,the parameter generation unit 53 can estimate a section in which 3D dataare lost by using a condition indicating that the two center axes of thetwo cylinders in the straight pipe part are parallel to each other.

The display control unit 56 displays a region R42 and a region R43 inthe region R41. The region R42 corresponds to the first connectionregion, and the region R43 corresponds to the second connection region.

There is a case in which the second 3D data include only the 3D data inthe circular arc part. In such a case, it is difficult for the parametergeneration unit 53 to generate a position-and-posture conversionparameter used for accurately connecting the first 3D data and thesecond 3D data together. The parameter generation unit 53 may generate atemporary position-and-posture conversion parameter such that a regionbetween the first 3D data and the second 3D data disappears. After animage of 3D data of a wide range is displayed, a user may adjust theposition and the posture of the 3D data as in the first modified exampleof the first embodiment. At this time, the user may adjust the positionof the 3D data along a circular arc.

Fifth Modified Example of First Embodiment

A fifth modified example of the first embodiment of the presentinvention will be described. In the fifth modified example of the firstembodiment, the image display device 50 shown in FIG. 1 is used.

A subject in the second to fourth modified examples of the firstembodiment described above is an inspection portion of a pipe. On theother hand, in the fifth modified example of the first embodiment, anexample in which a subject is an inspection portion of a gas turbinewill be described. The gas turbine is a main inspection target in anaircraft engine, an electrical generator, and the like. Hereinafter, anexample in a blade inspection will be described. A blade is a typicalinspection portion in an aircraft engine.

The size of a turbine blade of an electrical generator is different fromthat of a gas turbine blade of an aircraft engine. However, thestructure of the turbine blade is similar to that of the gas turbineblade. Therefore, the following processing may be applied to aninspection of the gas turbine blade.

A gas turbine includes two or more blades. The two or more blades arecircularly and radially fixed to a discoid component called a disk.

FIG. 18 schematically shows a configuration of a blade. In FIG. 18 ,twelve blades BL50 are disposed in a disk DS50. In fact, several tens ofblades or more than 100 blades are disposed in one disk. A centerposition CP50 indicates the center of the disk DS50 in a perpendicularplane to a rotation axis of an engine. Multiple steps of disks aredisposed in each of a compressor section and a turbine section of a gasturbine. A disk of each step includes many blades functioning as one ofa rotor blade and a stator blade. In many cases, an ID is allocated toeach blade.

When rotor blades are inspected, a scope is inserted into an engine froman access port provided in the engine. The rotor blades manually orautomatically rotate, and the distal end of the scope is fixed. Aninspector performs an inspection in this state and checks whether thereis an abnormality in each blade.

On the other hand, when stator blades (nozzle guide vanes) disposedimmediately behind a combustion chamber are inspected, the scope isinserted from an access port of the combustion chamber toward the rearof the combustion chamber. Hereinafter, nozzle guide vanes are calledstator blades. In an inspection of the stator blades, it is impossibleto rotate the blades. An inspector moves the distal end of the scopeabove the periphery of the disk of the stator blades and checks whetherthere is an abnormality in each blade.

The information INF10 of the dialog box DL10 shown in FIG. 4 includesinformation of a gas turbine, which is an inspection target, andinformation of blades in the gas turbine. For example, a user selects a“gas turbine” and then selects a “blade.” When the user inputsinformation indicating that the selection is finalized into the imagedisplay device 50, the information acceptance unit 57 accepts theinformation selected by the user. At this time, the informationacceptance unit 57 accepts a character string “blade.”

For example, structure information of blades indicates that blades inthe first 3D data and the second 3D data are disposed at regularintervals on a periphery of a circle having an arbitrary curvatureradius. The structure information indicates that the lengths of theblades are the same. The parameter generation unit 53 generates aconversion parameter on the basis of the structure information in StepS106.

The total number of blades, the length of each blade, and the ID of eachblade are known in a case in which an engine type, a section, and astage step number have already been identified. The engine typeindicates a type of an engine in which blades are disposed. The sectionindicates a compressor section or a turbine section. The stage stepnumber indicates a position (step number) of a disk in a stage includingmultiple steps of disks. The structure information may include the totalnumber of blades, the length of each blade, and the ID of each blade. Inthis case, the parameter generation unit 53 can generate aposition-and-posture conversion parameter used for accurately connectingthe first 3D data and the second 3D data together.

The image display device 50 executes the processing shown in FIG. 2 .Specific processing in Step S106 is different from that in a case inwhich an inspection target is a pipe.

The parameter generation unit 53 executes processing shown in FIG. 19 inStep S106. FIG. 19 shows a procedure of the processing executed by theparameter generation unit 53. In the following example, the parametergeneration unit 53 generates a position-and-posture conversion parameterused for causing the position and the posture of the second 3Dcoordinate system to match the position and the posture of the first 3Dcoordinate system, respectively. In addition, the parameter generationunit 53 generates a scale conversion parameter used for causing thescale of the second 3D coordinate system to match the scale of the first3D coordinate system.

The structure information indicates that the lengths of the blades arethe same. Therefore, the parameter generation unit 53 generates a scaleconversion parameter used for correcting the scale of the second 3D datasuch that the length of the blade of the first 3D data and the length ofthe blade of the second 3D data match each other (Step S106 e). Thestructure information may include information indicating the totallength of the blade. The parameter generation unit 53 may determine thatthe length of the blade of the first 3D data and the length of the bladeof the second 3D data are the same on the basis of the information.

After Step S106 e, the parameter generation unit 53 calculates a centerposition (first center position) of the disk in the first 3D data (StepS106 f). In the example shown in FIG. 18 , the parameter generation unit53 calculates the center position CP50.

The parameter generation unit 53 may use any method in order tocalculate a center position of the disk. For example, the parametergeneration unit 53 may calculate a position at which straight lines,each of which extends in a longitudinal direction of each blade,intersect as a center position. The parameter generation unit 53 maydetect a blade in 3D data by executing pattern matching. The parametergeneration unit 53 may calculate a circumscribed or inscribed circle onthe basis of a positional relationship between the blades and maycalculate a center position of the circle.

After Step S106 f, the parameter generation unit 53 calculates a centerposition (second center position) of the disk in the second 3D data byexecuting similar processing to Step S106 f (Step S106 g). A method ofcalculating the center position of the disk in Step S106 g may be thesame as or different from that of calculating the center position of thedisk in Step S106 f.

After Step S106 g, the parameter generation unit 53 generates aposition-and-posture conversion parameter used for correcting thepositions of the first 3D data and the second 3D data such that thefirst center position and the second center position match each other(Step S106 h).

After Step S106 h, the parameter generation unit 53 calculates aposition-and-posture conversion parameter used for disposing the startend blade in the second 3D data as a next blade to the terminal endblade in the first 3D data (Step S106 i).

When Step S106 i is executed, the processing shown in FIG. 19 iscompleted.

FIG. 20 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG51 on the display unit71. The image IMG51 includes a region R51. An image of a first 3D shape3D51 and a second 3D shape 3D52 is displayed in the region R51. Thefirst 3D shape 3D51 is indicated by the first 3D data. The second 3Dshape 3D52 is indicated by the second 3D data.

The first 3D shape 3D51 includes four blades. The first 3D data includetimestamps from a time point t1 to a time point t2 shown in FIG. 20 . Atimestamp (t1) of a blade BL51 is the earliest, and a timestamp (t2) ofa blade BL52 is the last. The second 3D shape 3D52 includes eightblades. The second 3D data include timestamps from a time point t3 to atime point t4 shown in FIG. 20 . A timestamp (t3) of a blade BL53 is theearliest, and a timestamp (t4) of a blade BL54 is the last.

The parameter generation unit 53 identifies a first connection region ofthe first 3D shape 3D51 associated with the time point t2 and identifiesa second connection region of the second 3D shape 3D52 associated withthe time point t3. The first connection region includes the blade BL52at the terminal end of the first 3D shape 3D51. The second connectionregion includes the blade BL53 at the start end of the second 3D shape3D52. The parameter generation unit 53 generates a position-and-postureconversion parameter used for connecting the second connection region tothe first connection region.

The parameter generation unit 53 identifies a first connection region ofthe first 3D shape 3D51 associated with the time point t1 and identifiesa second connection region of the second 3D shape 3D52 associated withthe time point t4. The first connection region includes the blade BL51at the start end of the first 3D shape 3D51. The second connectionregion includes the blade BL54 at the terminal end of the second 3Dshape 3D52. The parameter generation unit 53 generates aposition-and-posture conversion parameter used for connecting the secondconnection region to the first connection region.

The display control unit 56 displays a region R52 and a region R53 inthe region R51. Each of the region R52 and the region R53 includes thefirst connection region and the second connection region.

The positional relationship between the blade BL51 and the blade BL54may be determined on the basis of features of two or more images inorder to connect the first connection region including the blade BL51and the second connection region including the blade BL54 together. Thetwo or more images are used for generating the first 3D data and thesecond 3D data. The first connection region and the second connectionregion may be connected together on the basis of the features. Thismethod is called loop closing. A known technique can be applied to theloop closing.

In a case in which the total number of actual blades and the totalnumber of blades in 3D data are the same, the parameter generation unit53 can generate a position-and-posture conversion parameter used foraccurately connecting the first 3D data and the second 3D data together.The total number of blades in the 3D data is equal to the sum of thenumber of blades in the first 3D data and the number of blades in thesecond 3D data.

FIG. 21 shows an example of an image displayed on the display unit 71when the total number of blades in the 3D data is less than that ofactual blades by one. The same parts as those shown in FIG. 20 will notbe described.

A second 3D shape 3D52 shown in FIG. 21 does not include the blade BL54shown in FIG. 20 . There is a possibility that the second 3D shape 3D52shown in FIG. 21 does not include the blade BL53 shown in FIG. 20 .Therefore, there is a possibility that the second 3D shape 3D52 does notactually include one of the blade BL53 and the blade BL54.

After an image of 3D data of a wide range is displayed, a user mayadjust the position and the posture of the 3D data as in the firstmodified example of the first embodiment. At this time, the user mayadjust the position of the second 3D data by rotating the second 3Ddata. The user may rotate the first 3D data and the second 3D data. Theposition of the 3D data may be adjusted only on a periphery of a circlehaving an arbitrary diameter.

Second Embodiment

A second embodiment of the present invention will be described. In thesecond embodiment, a method of generating the first 3D data and thesecond 3D data will be described. Hereinafter, an example in which theimage display device is an endoscope will be described.

A configuration of an endoscope device 1 in the second embodiment willbe described by referring to FIG. 22 and FIG. 23 . FIG. 22 shows anexternal appearance of the endoscope device 1. FIG. 23 shows an internalconfiguration of the endoscope device 1.

The endoscope device 1 shown in FIG. 22 includes the insertion unit 2, amain body unit 3, an operation unit 4, and a display unit 5. Theendoscope device 1 images a subject and generates an image. The subjectis an industrial product. In order to observe various subjects, a usercan perform replacement of an optical adaptor mounted at a distal end 20of the insertion unit 2, selection of a built-in video-processingprogram, and addition of a video-processing program.

The insertion unit 2 is inserted inside a subject. The insertion unit 2has a long and thin bendable tube shape from the distal end 20 to a baseend portion. The insertion unit 2 images a subject and outputs animaging signal to the main body unit 3. An optical adapter is mounted onthe distal end 20 of the insertion unit 2. For example, a single-eyeoptical adapter is mounted on the distal end 20. The main body unit 3 isa control device including a housing unit that houses the insertion unit2. The operation unit 4 accepts an operation for the endoscope device 1from a user. The display unit 5 includes a display screen and displaysan image of a subject acquired by the insertion unit 2, an operationmenu, and the like on the display screen.

The operation unit 4 is a user interface. The display unit 5 is amonitor (display) such as a liquid crystal display (LCD). The displayunit 5 may be a touch panel. In such a case, the operation unit 4 andthe display unit 5 are integrated.

The main body unit 3 shown in FIG. 23 includes an endoscope unit 8, acamera control unit (CCU) 9, and a control device 10.

The endoscope unit 8 includes a light source device and a bending devicenot shown in the drawing. The light source device provides the distalend 20 with illumination light that is necessary for observation. Thebending device bends a bending mechanism that is built in the insertionunit 2.

An imaging device 28 is built in the distal end 20 of the insertion unit2. The imaging device 28 is an image sensor. The imaging device 28photo-electrically converts an optical image of a subject formed by anoptical adaptor and generates an imaging signal.

The CCU 9 drives the imaging device 28. An imaging signal output fromthe imaging device 28 is input into the CCU 9. The CCU 9 performspre-processing including amplification, noise elimination, and the likefor the imaging signal acquired by the imaging device 28. The CCU 9converts the imaging signal for which the pre-processing has beenexecuted into a video signal such as an NTSC signal.

The control device 10 includes a video-signal-processing circuit 12, aread-only memory (ROM) 13, a random-access memory (RAM) 14, a cardinterface 15, an external device interface 16, a control interface 17,and a central processing unit (CPU) 18.

The video-signal-processing circuit 12 performs predetermined videoprocessing on the video signal output from the CCU 9. For example, thevideo-signal-processing circuit 12 performs video processing related toimprovement of visibility. For example, the video processing is colorreproduction, gray scale correction, noise suppression, contourenhancement, and the like. For example, the video-signal-processingcircuit 12 combines the video signal output from the CCU 9 and a graphicimage signal generated by the CPU 18. The graphic image signal includesan image of the operation screen and the like. Thevideo-signal-processing circuit 12 outputs a combined video signal tothe display unit 5.

The ROM 13 is a nonvolatile recording medium on which a program for theCPU 18 to control the operation of the endoscope device 1 is recorded.The RAM 14 is a volatile recording medium that temporarily storesinformation used by the CPU 18 for controlling the endoscope device 1.The CPU 18 controls the operation of the endoscope device 1 on the basisof the program recorded on the ROM 13.

A memory card 42 is connected to the card interface 15. The memory card42 is a recording medium that is attachable to and detachable from theendoscope device 1. The card interface 15 inputs control-processinginformation, image information, and the like stored on the memory card42 into the control device 10. In addition, the card interface 15records the control-processing information, the image information, andthe like generated by the endoscope device 1 on the memory card 42.

An external device such as a USB device is connected to the externaldevice interface 16. For example, a personal computer (PC) 41 isconnected to the external device interface 16. The external deviceinterface 16 transmits information to the PC 41 and receives informationfrom the PC 41. In this way, the PC 41 can display information. Inaddition, by inputting an instruction into the PC 41, a user can performan operation related to control of the endoscope device 1.

The control interface 17 performs communication with the operation unit4, the endoscope unit 8, and the CCU 9 for operation control. Thecontrol interface 17 notifies the CPU 18 of an instruction input intothe operation unit 4 by the user. The control interface 17 outputscontrol signals used for controlling the light source device and thebending device to the endoscope unit 8. The control interface 17 outputsa control signal used for controlling the imaging device 28 to the CCU9.

A program executed by the CPU 18 may be recorded on a computer-readablerecording medium. The program recorded on this recording medium may beread and executed by a computer other than the endoscope device 1. Forexample, the program may be read and executed by the PC 41. The PC 41may control the endoscope device 1 by transmitting control informationused for controlling the endoscope device 1 to the endoscope device 1 inaccordance with the program. Alternatively, the PC 41 may acquire avideo signal from the endoscope device 1 and may process the acquiredvideo signal.

The imaging device 28 is a camera that acquires a still image group. Thestill image group includes two or more images. Each of the two or moreimages is temporally associated with the other images included in thetwo or more images. For example, each of the two or more images is astill image. A video may be used instead of the still image group. Twoor more frames included in the video are associated with each other bytimestamps (timecodes).

For example, the imaging device 28 is a single-eye camera having asingle viewpoint. In this case, each of two or more still images is animage acquired by the single-eye camera. A camera in the secondembodiment includes the imaging device 28 and an observation opticalsystem.

As described above, the endoscope device 1 includes the imaging device28 and the CPU 18. The imaging device 28 images a subject and generatesan imaging signal. The imaging signal includes an image of the subject.Accordingly, the imaging device 28 acquires the image of the subjectgenerated by imaging the subject. The image is a two-dimensional image(2D image). The image acquired by the imaging device 28 is input intothe CPU 18 via the video-signal-processing circuit 12.

FIG. 24 shows a functional configuration of the CPU 18. The CPU 18 hasfunctional units including a control unit 180, a data acquisition unit181, a parameter generation unit 182, a conversion unit 183, a datageneration unit 184, a display control unit 185, an informationacceptance unit 186, a structure estimation unit 187, and a datageneration unit 188. At least one of the blocks shown in FIG. 24 may beconstituted by a different circuit from the CPU 18.

Each unit shown in FIG. 24 may be constituted by at least one of aprocessor and a logic circuit. Each unit shown in FIG. 24 may includeone or a plurality of processors. Each unit shown in FIG. 24 may includeone or a plurality of logic circuits.

The control unit 180 controls processing executed by each unit shown inFIG. 24 .

The data acquisition unit 181 has a similar function to that of the dataacquisition unit 52 shown in FIG. 1 . The data acquisition unit 181connects to the RAM 14 and acquires first 3D data and second 3D datafrom the RAM 14. The data acquisition unit 181 may acquire the first 3Ddata and the second 3D data from a recording medium in the PC 41 or fromthe memory card 42.

The parameter generation unit 182 has a similar function to that of theparameter generation unit 53 shown in FIG. 1 . The parameter generationunit 182 generates a conversion parameter used for converting the first3D coordinate system and the second 3D coordinate system into a commoncoordinate system.

The conversion unit 183 has a similar function to that of the conversionunit 54 shown in FIG. 1 . The conversion unit 183 converts the first 3Dcoordinate system and the second 3D coordinate system into the commoncoordinate system by using the conversion parameter generated by theparameter generation unit 182. In this way, the conversion unit 183converts the first 3D data and the second 3D data into 3D data in thecommon coordinate system.

The data generation unit 184 has a similar function to that of the datageneration unit 55 shown in FIG. 1 . The data generation unit 184connects together the first 3D data and the second 3D data converted bythe conversion unit 183 into the 3D data in the common coordinatesystem. In this way, the data generation unit 184 generates 3D data of awide range of a subject.

The display control unit 185 controls processing executed by thevideo-signal-processing circuit 12. The CCU 9 outputs a video signal.The video signal includes color data of each pixel of an image acquiredby the imaging device 28. The display control unit 185 causes thevideo-signal-processing circuit 12 to output the video signal outputfrom the CCU 9 to the display unit 5. The video-signal-processingcircuit 12 outputs the video signal to the display unit 5. The displayunit 5 displays an image on the basis of the video signal output fromthe video-signal-processing circuit 12. In this way, the display controlunit 185 displays the image acquired by the imaging device 28 on thedisplay unit 5.

The display control unit 185 displays various kinds of information onthe display unit 5. In other words, the display control unit 185displays various kinds of information on an image. The various kinds ofinformation may include a cursor. The cursor is a mark used by a user todesignate a specific point on an image.

For example, the display control unit 185 generates a graphic imagesignal of the various kinds of information. The display control unit 185outputs the generated graphic image signal to thevideo-signal-processing circuit 12. The video-signal-processing circuit12 combines the video signal output from the CCU 9 and the graphic imagesignal output from the CPU 18. In this way, the various kinds ofinformation are superimposed on an image. The video-signal-processingcircuit 12 outputs the combined video signal to the display unit 5. Thedisplay unit 5 displays an image on which the various kinds ofinformation are superimposed.

In addition, the display control unit 185 has a similar function to thatof the display control unit 56 shown in FIG. 1 . The display controlunit 185 generates a graphic image signal of 3D data. The displaycontrol unit 185 outputs the graphic image signal to thevideo-signal-processing circuit 12. Similar processing to that describedabove is executed, and the display unit 5 displays an image of the 3Ddata. In this way, the display control unit 185 displays the image ofthe 3D data on the display unit 5.

A user inputs various kinds of information into the endoscope device 1by operating the operation unit 4. The operation unit 4 outputs theinformation input by the user. The information is input into the controlinterface 17, which is an input unit. The information is output from thecontrol interface 17 to the CPU 18. The information acceptance unit 186accepts the information input into the endoscope device 1 via theoperation unit 4.

For example, a user inputs position information of a cursor into theendoscope device 1 by operating the operation unit 4. In a case in whichthe display unit 5 is constituted as a touch panel, the user inputsposition information indicating a position on an image into theendoscope device 1 by touching the screen of the display unit 5. Theinformation acceptance unit 186 accepts the position information inputinto the endoscope device 1. The information acceptance unit 186calculates the position on the image on the basis of the positioninformation. The display control unit 185 displays a cursor at theposition calculated by the information acceptance unit 186.

In addition, the information acceptance unit 186 has a similar functionto that of the information acceptance unit 57 shown in FIG. 1 . Theinformation acceptance unit 186 accepts structure information.

The structure estimation unit 187 has a similar function to that of thestructure estimation unit 58 shown in FIG. 1 . In a case in which theinformation acceptance unit 186 accepts information not includingstructure information, the structure estimation unit 187 estimates astructure of a subject in a connection region on the basis of theaccepted information and generates structure information.

The data generation unit 188 has functions shown in FIG. 25 . FIG. 25shows a functional configuration of the data generation unit 188. Thedata generation unit 188 has functional units including an imageacquisition unit 1880, a condition acceptance unit 1881, and a datacalculation unit 1882.

The image acquisition unit 1880 acquires the still image group from theRAM 14. The image acquisition unit 1880 may acquire the still imagegroup from a recording medium in the PC 41 or from the memory card 42.

A user inputs information indicating a condition for generating 3D datainto the endoscope device 1 by operating the operation unit 4. Thecondition acceptance unit 1881 accepts the condition for generating the3D data on the basis of the information input by the user. Specifically,the condition includes an internal parameter of a camera, a distortioncorrection parameter of the camera, a setting value, a reference length,and the like. The setting value is used for various kinds of processingfor generating the 3D data. The reference length is used for matchingthe 3D data with the actual scale of a subject.

The data calculation unit 1882 generates (reconfigures) 3D data of asubject on the basis of two or more images included in the still imagegroup. The data calculation unit 1882 does not need to use all theimages included in the still image group. In a case in which the stillimage group includes three or more images, the data calculation unit1882 generates the 3D model on the basis of all or part of the stillimage group. The 3D data include 3D coordinates of two or more points(three-dimensional point cloud) of the subject, a camera coordinate, andposture information.

The camera coordinate indicates 3D coordinates of a camera, whichacquires each of the two or more images, and is associated with each ofthe two or more images. The camera coordinate indicates 3D coordinatesof a viewpoint when an image is acquired. For example, the cameracoordinate indicates 3D coordinates of an observation optical systemincluded in the camera. The posture information indicates a posture ofthe camera, which acquires each of the two or more images, and isassociated with each of the two or more images. For example, the postureinformation indicates a posture of the observation optical systemincluded in the camera.

A procedure of specific processing executed by the data calculation unit1882 will be described. The data calculation unit 1882 uses the stillimage group acquired by the image acquisition unit 1880 and thecondition accepted by the condition acceptance unit 1881. Hereinafter,an example in which the data calculation unit 1882 uses two images(still images) included in the still image group will be described. Whenthe two images are generated, two viewpoints of the camera are differentfrom each other. Even when three or more images are used, a basicprinciple is not changed from that of a case in which two images areused. A method described below may be also applied to a case in whichthree or more images are used.

In the method described below, feature points in each of two imagesacquired at two different viewpoints are detected, and a plurality offeature points are associated with each other. In addition, in themethod described below, the position of the camera, the posture of thecamera, and the 3D coordinates of the feature points are estimated onthe basis of the plurality of feature points. A method using informationof feature points is called an indirect method. A method applied to eachembodiment of the present invention is not limited to this method.

For example, there is a method of directly using pixel values of twoimages acquired at two different viewpoints. By using this method, theposition of the camera, the posture of the camera, and the 3Dcoordinates corresponding to each pixel are estimated. This method iscalled a direct method. This method may be used in each embodiment ofthe present invention. As long as the position of the camera, theposture of the camera, and the 3D coordinates of a subject are estimatedby using two or more images acquired at two or more differentviewpoints, any method may be used.

FIG. 26 schematically shows a situation of image acquisition in a casein which two images of a subject are acquired. In the followingdescription, an expression “camera” is used in a broad sense. In a casein which an endoscope acquires an image, the camera in the followingdescription specifically indicates an observation optical system of thedistal end of the endoscope.

As shown in FIG. 26 , first, an image I₁ is acquired in an imaging statec₁ of the camera. Next, an image I₂ is acquired in an imaging state c₂of the camera. At least one of an imaging position and an imagingposture is different between the imaging state c₁ and the imaging statec₂. In FIG. 26 , both the imaging position and the imaging posture aredifferent between the imaging state c₁ and the imaging state c₂.

In each embodiment of the present invention, it is assumed that theimage I₁ and the image I₂ are acquired by the same endoscope. Inaddition, in each embodiment of the present invention, it is assumedthat parameters of an objective optical system of the endoscope do notchange. The parameters of the objective optical system are a focaldistance, a distortion aberration, a pixel size of an image sensor, andthe like. Hereinafter, for the convenience of description, theparameters of the objective optical system will be abbreviated tointernal parameters. When such conditions are assumed, the internalparameters specifying characteristics of the optical system of theendoscope can be used in common regardless of the position and theposture of the camera (observation optical system). In each embodimentof the present invention, it is assumed that the internal parameters areacquired at the time of factory shipment. In addition, in eachembodiment of the present invention, it is assumed that the internalparameters are known at the time of acquiring an image.

In each embodiment of the present invention, it is assumed that two ormore images are extracted from the still image group and the still imagegroup is acquired by one endoscope. However, the present invention isnot limited to this. For example, the present invention may be alsoapplied to a case in which a 3D model is restored by using a pluralityof still image groups acquired by a plurality of endoscopes. In thiscase, the image I₁ and the image I₂ have only to be acquired by usingdifferent endoscope devices, and each internal parameter has only to bestored for each endoscope. Even if the internal parameters are unknown,it is possible to perform calculation by using the internal parametersas variables. Therefore, the subsequent procedure does not greatlychange in accordance with whether the internal parameters are known.

Processing for calculating 3D coordinates of a subject on the basis oftwo images and generating 3D data will be described by using FIG. 27 .FIG. 27 shows a procedure of processing for generating 3D data.

First, the data calculation unit 1882 executes feature point detectionprocessing (Step S200). The data calculation unit 1882 detects a featurepoint of each of two images in the feature point detection processing.The feature point indicates a corner, an edge, and the like in which animage luminance gradient is large in information of a subject seen in animage. As a method of detecting this feature point, scale-invariantfeature transform (SIFT), features from accelerated segment test (FAST),or the like is used. The data calculation unit 1882 can detect a featurepoint within an image by using such a method.

FIG. 26 shows an example in which a feature point Pu is detected fromthe image I₁ and a feature point P₁₂ is detected from the image I₂.Although only one feature point of each image is shown in FIG. 26 , infact, a plurality of feature points are detected in each image. There isa possibility that the number of feature points detected in each imageis different between images. Each feature point detected from each imageis converted into data called a feature quantity. The feature quantityis data indicating characteristics of a feature point.

After Step S200, the data calculation unit 1882 executes feature pointassociation processing (Step S201). In the feature point associationprocessing, the data calculation unit 1882 compares correlations offeature quantities between images for each feature point detected in thefeature point detection processing (Step S200). In a case in which thecorrelations of the feature quantities are compared and a feature pointof which feature quantities are close to those of a feature point ofanother image is found in each image, the data calculation unit 1882stores information of the feature point on the RAM 14. In this way, thedata calculation unit 1882 associates feature points of respectiveimages together. On the other hand, in a case in which a feature pointof which feature quantities are close to those of a feature point ofanother image is not found, the data calculation unit 1882 discardsinformation of the feature point.

After Step S201, the data calculation unit 1882 reads coordinates offeature points (feature point pair) of two images associated with eachother from the RAM 14. The data calculation unit 1882 executesprocessing of calculating a position and a posture on the basis of theread coordinates (Step S202). In the processing of calculating aposition and a posture, the data calculation unit 1882 calculates arelative position and a relative posture between the imaging state c₁ ofthe camera that acquires the image I₁ and the imaging state c₂ of thecamera that acquires the image I₂. More specifically, the datacalculation unit 1882 calculates a matrix E by solving the followingEquation (1) using an epipolar restriction.

$\begin{matrix}{{p_{1}^{T}{Ep}_{2}} = {{0E} = {{{\lbrack t\rbrack_{X}R}\because\lbrack t\rbrack_{X}} = \begin{pmatrix}0 & {- t_{z}} & t_{y} \\t_{z} & 0 & {- t_{x}} \\{- t_{y}} & t_{x} & 0\end{pmatrix}}}} & (1)\end{matrix}$

The matrix E is called a basic matrix. The basic matrix E is a matrixstoring a relative position and a relative posture between the imagingstate c₁ of the camera that acquires the image I₁ and the imaging statec₂ of the camera that acquires the image I₂. In Equation (1), a matrixp₁ is a matrix including coordinates of a feature point detected fromthe image I_(t). A matrix p₂ is a matrix including coordinates of afeature point detected from the image I₂. The basic matrix E includesinformation related to a relative position and a relative posture of thecamera and thus corresponds to external parameters of the camera. Thedata calculation unit 1882 can solve the basic matrix E by using a knownalgorithm.

As shown in FIG. 26 , Expression (2) and Expression (3) are satisfied ina case in which the amount of position (relative position) change of thecamera is t and the amount of posture (relative posture) change of thecamera is R.

$\begin{matrix}{t = \left( {t_{x},t_{y},t_{z}} \right)} & (2)\end{matrix}$ $\begin{matrix}{R = {{{R_{x}(\alpha)}{R_{y}(\beta)}{R_{z}(\gamma)}} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\alpha} & {{- s}{in}\alpha} \\0 & {\sin\alpha} & {\cos\alpha}\end{pmatrix}\begin{pmatrix}{\cos\beta} & 0 & {\sin\beta} \\0 & 1 & 0 \\{{- s}{in}\beta} & 0 & {\cos\beta}\end{pmatrix}\begin{pmatrix}{\cos\gamma} & {{- s}{in}\gamma} & 0 \\{\sin\gamma} & {\cos\gamma} & 0 \\0 & 0 & 1\end{pmatrix}}}} & (3)\end{matrix}$

In Expression (2), the amount of movement in an x-axis direction isexpressed as t_(x), the amount of movement in a y-axis direction isexpressed as t_(y), and the amount of movement in a z-axis direction isexpressed as t_(z). In Expression (3), a rotation amount α around thex-axis is expressed as R_(x)(α), a rotation amount β around the y axisis expressed as R_(y)(β), and a rotation amount γ around the z axis isexpressed as R_(z)(γ). After the basic matrix E is calculated,optimization processing called bundle adjustment may be executed inorder to improve restoration accuracy of 3D coordinates.

The data calculation unit 1882 calculates 3D coordinates (cameracoordinate) in a coordinate system of a 3D model by using the calculatedamount of positional change of the camera. For example, the datacalculation unit 1882 defines 3D coordinates of the camera that acquiresthe image I₁. The data calculation unit 1882 calculates 3D coordinatesof the camera that acquires the image I₂ on the basis of the 3Dcoordinates of the camera that acquires the image I₁ and the amount ofpositional change of the camera that acquires the image I₂.

The data calculation unit 1882 calculates posture information in acoordinate system of a 3D model by using the calculated amount ofpostural change of the camera. For example, the data calculation unit1882 defines posture information of the camera that acquires the imageI₁. The data calculation unit 1882 generates posture information of thecamera that acquires the image I₂ on the basis of the postureinformation of the camera that acquires the image I₁ and the amount ofpostural change of the camera that acquires the image I₂.

The data calculation unit 1882 generates data (3D shape data) of athree-dimensional shape (3D shape) by executing the processing (StepS202) of calculating a position and a posture. The 3D shape data include3D coordinates (camera coordinate) at a position of a camera and postureinformation indicating a posture of the camera. In addition, in a casein which a method such as structure from motion, visual-SLAM, or thelike is applied to the processing (Step S202) of calculating a positionand a posture, the data calculation unit 1882 further calculates 3Dcoordinates of each feature point in Step S202. The 3D shape datagenerated in Step S202 do not include 3D coordinates of points on asubject other than the feature point. Therefore, the 3D shape dataindicate a sparse 3D shape of the subject.

The 3D shape data include the 3D coordinates of each feature point, theabove-described camera coordinate, and the above-described postureinformation. The 3D coordinates of each feature point are defined in thecoordinate system of the 3D data.

The 3D coordinates of each feature point are associated withtwo-dimensional coordinates (2D coordinates) of each feature point. The2D coordinates of each feature point are defined in a coordinate systemof an image including each feature point. The 2D coordinates and the 3Dcoordinates of each feature point are associated with an image includingeach feature point.

After Step S202, the data calculation unit 1882 executes processing ofrestoring a three-dimensional shape on the basis of the relativeposition and the relative posture of the camera (the amount t ofpositional change and the amount R of postural change) calculated inStep S202 (Step S203). The data calculation unit 1882 generates 3D dataof a subject in the processing of restoring a three-dimensional shape.As a technique for restoring a three-dimensional shape of a subject,there are patch-based multi-view stereo (PMVS), matching-processing thatuses parallelization stereo, and the like. However, a means therefor isnot particularly limited.

The data calculation unit 1882 calculates 3D coordinates of points on asubject other than feature points in Step S203. The 3D coordinates ofeach point other than the feature points are defined in the coordinatesystem of the 3D data. The 3D coordinates of each point are associatedwith the 2D coordinates of each point. The 2D coordinates of each pointare defined in a coordinate system of a 2D image including each point.The 3D coordinates and the 2D coordinates of each point are associatedwith a 2D image including each point. The data calculation unit 1882updates the 3D shape data. The updated 3D shape data include the 3Dcoordinates of each feature point, the 3D coordinates of each pointother than the feature points, the camera coordinate, and the postureinformation. The 3D shape data updated in Step S203 include 3Dcoordinates of a point on the subject other than the feature points inaddition to the 3D coordinates of the feature points. Therefore, the 3Dshape data indicate a dense 3D shape of the subject.

After Step S203, the data calculation unit 1882 executesthree-dimensional coordinate transformation processing on the basis ofboth the 3D shape data processed in the processing (Step S203) ofrestoring a three-dimensional shape and the reference length accepted bythe condition acceptance unit 1881 (Step S204). The data calculationunit 1882 transforms the 3D shape data of the subject intothree-dimensional coordinate data (3D data) having a dimension of lengthin the three-dimensional coordinate transformation processing. When StepS204 is executed, the processing shown in FIG. 27 is completed.

In order to shorten a processing time, Step S203 may be omitted. In thiscase, after Step S202 is executed, Step S204 is executed withoutexecuting Step S203.

Step S204 may be omitted. In this case, after Step S203 is executed, theprocessing shown in FIG. 27 is completed without executing Step S204. Inthis case, the 3D data indicate a relative shape of the subject nothaving a dimension of length. Even when the 3D data indicate a relativeshape of the subject, the endoscope device 1 can identify a region ofthe 3D data corresponding to the camera coordinate.

It is necessary that at least part of a region of one of images and atleast part of a region of each of at least one of the other imagesoverlap each other in order to generate 3D data in accordance with theprinciple shown in FIG. 26 . In other words, a region of a first imageand a region of a second image different from the first image include acommon region. The other region in the first image excluding the commonregion and the other region in the second image excluding the commonregion are different from each other.

After the 3D data are generated, the endoscope device 1 executes similarprocessing to that shown in FIG. 2 . The endoscope device 1 can connectthe first 3D data and the second 3D data together and can display animage of a 3D shape of a wide range. The image display device 50 shownin FIG. 1 may include the data generation unit 188.

Third Embodiment

A third embodiment of the present invention will be described. In thethird embodiment, a structure in a connection region of a subject isestimated by using data generated by a sensor.

The image display device 50 shown in FIG. 1 is changed to an imagedisplay device 50 b shown in FIG. 28 . FIG. 28 shows a configuration ofthe image display device 50 b. The same configuration as that shown inFIG. 1 will not be described.

The image display device 50 b shown in FIG. 28 includes a control unit51, a data acquisition unit 52, a parameter generation unit 53, aconversion unit 54, a data generation unit 55, a display control unit56, and a structure estimation unit 58. The image display device 50 bdoes not include the information acceptance unit 57 shown in FIG. 1 . Anoperation unit 70, a display unit 71, a communication unit 72, a storageunit 73, and a sensor 74 shown in FIG. 28 are connected to the imagedisplay device 50 b. The image display device 50 b may include at leastone of the operation unit 70, the display unit 71, the communicationunit 72, the storage unit 73, and the sensor 74.

Each unit of the image display device 50 b may be constituted by atleast one of a processor and a logic circuit. Each unit of the imagedisplay device 50 b may include one or a plurality of processors. Eachunit of the image display device 50 b may include one or a plurality oflogic circuits.

The sensor 74 outputs sensor data. The sensor data are influenced by ageometric structure of a subject. The sensor 74 is an image sensor(imaging device).

The sensor data include a pixel value, a measured value of anacceleration, or a measured value of an angular velocity as a sensorvalue. The image sensor generates an image of a subject. The structureof the subject is seen in the image.

The sensor data are saved on the storage unit 73. The structureestimation unit 58 estimates a structure of a subject in a connectionregion by using the sensor data and generates structure information.

Processing executed by the image display device 50 b will be describedby using FIG. 29 . FIG. 29 shows a procedure of the processing executedby the image display device 50 b. The same processing as that shown inFIG. 2 will not be described.

After Step S103, the data acquisition unit 52 connects to the storageunit 73 and acquires the sensor data in the connection region from thestorage unit 73 (Step S120).

The 3D data include timestamps. In the sensor data, a sensor value and atimestamp are associated with each other. Therefore, the dataacquisition unit 52 can acquire sensor data generated at a time pointindicated by the timestamp of the 3D data. For example, the dataacquisition unit 52 acquires sensor data in a section from the timepoint of the terminal end of the first 3D data to the time point of thestart end of the second 3D data from the storage unit 73.

Hereinafter, an example in which the sensor 74 is an image sensor andthe data acquisition unit 52 acquires two or more images from thestorage unit 73 will be described. The following processing can be alsoapplied to a case in which the data acquisition unit 52 acquires oneimage from the storage unit 73.

After Step S120, the structure estimation unit 58 estimates a structureof a subject in a connection region by using the two or more imagesacquired in Step S120 and generates structure information (Step S121).After Step S121, Step S106 is executed.

The structure estimation unit 58 may use any method in order to estimatethe structure of the subject. For example, the structure estimation unit58 may estimate the structure of the subject by using AI. The structureestimation unit 58 may estimate the structure of the subject bydetermining features of an image, calculating feature quantities, andusing support vector machines (SVM).

When the structure estimation unit 58 determines that the subject is astraight pipe, the structure estimation unit 58 may acquire structureinformation of the straight pipe from the reference table TB11 shown inFIG. 5 .

Each aspect of the present invention may include the following modifiedexample. The structure information is generated on the basis of dataoutput from the sensor 74.

In the third embodiment, the structure estimation unit 58 estimates astructure of a subject in a connection region by using sensor data. Theimage display device 50 b does not need to accept information from auser.

First Modified Example of Third Embodiment

A first modified example of the third embodiment of the presentinvention will be described. In the first modified example of the thirdembodiment, the image display device 50 b shown in FIG. 28 is used. Inthe first modified example of the third embodiment, the endoscope device1 shown in FIG. 22 is used in order to generate 3D data.

In the third embodiment described above, the structure estimation unit58 estimates a structure of a subject in a connection region by using animage generated by an image sensor. On the other hand, in the firstmodified example of the third embodiment, the structure estimation unit58 uses sensor data output from an inertial measurement unit (IMU)instead of using an image.

The IMU is used as the sensor 74. The sensor 74 includes an accelerationsensor and an angular velocity sensor and determines an acceleration andan angular velocity. For example, the sensor 74 is disposed in thedistal end 20 of the insertion unit 2. In a case in which the insertionunit 2 is inserted into a pipe, the insertion unit 2 moves in accordancewith the structure of the pipe. Therefore, the sensor 74 outputs sensordata in accordance with the structure of the pipe. The structureestimation unit 58 calculates traces (tracks) of the distal end 20 byusing the sensor data. The traces indicate two or more positions atwhich the distal end 20 is disposed.

The image display device 50 b executes the processing shown in FIG. 29 .The structure estimation unit 58 calculates traces of the distal end 20by using the sensor data in Step S121. The structure estimation unit 58estimates a structure of a subject in a connection region on the basisof the traces and generates structure information in Step S121.

For example, the structure estimation unit 58 estimates the structure ofthe subject by executing the following processing. The first 3D data andthe second 3D data include 3D coordinates in each of a first section, asecond section, and a third section. The first section includes thestart end of a 3D shape. The second section includes the terminal end ofthe 3D shape. The third section is disposed between the first sectionand the second section. For example, the second section in the first 3Ddata and the first section in the second 3D data correspond to aconnection region.

The structure estimation unit 58 analyzes the traces of the distal end20 in each of the second section in the first 3D data, the third sectionin the first 3D data close to the second section, the first section inthe second 3D data, and the third section in the second 3D data close tothe first section. When the structure estimation unit 58 determines thatthe distal end 20 moves almost linearly in all the sections, thestructure estimation unit 58 determines that the subject is a straightpipe. When the structure estimation unit 58 determines that the movingdirection of the distal end 20 changes by almost 90 degrees as thedistal end 20 passes through the above-described sections, the structureestimation unit 58 determines that the subject is a 90-degree fitting.The structure estimation unit 58 generates structure information inaccordance with the structure of the subject.

Second Modified Example of Third Embodiment

A second modified example of the third embodiment of the presentinvention will be described. In the second modified example of the thirdembodiment, the image display device 50 b shown in FIG. 28 is used. Theimage display device 50 b does not need to include the sensor 74. In thesecond modified example of the third embodiment, the endoscope device 1shown in FIG. 22 is used in order to generate 3D data.

In the third embodiment described above, the structure estimation unit58 estimates a structure of a subject in a connection region by using animage generated by an image sensor. On the other hand, in the secondmodified example of the third embodiment, the structure estimation unit58 uses history information of operations related to bending of theinsertion unit 2 instead of using an image.

A user inputs information indicating a bending direction and a bendingamount into the endoscope device 1 by operating the operation unit 4.The control unit 180 of the CPU 18 generates a control signal on thebasis of the information input by the user. The control signal is outputto the endoscope unit 8 via the control interface 17. The bending deviceof the endoscope unit 8 bends the bending mechanism on the basis of thecontrol signal.

The control unit 180 generates history information of operations relatedto bending of the insertion unit 2. The history information includesinformation indicating the bending direction and the bending amount. Thecommunication unit 72 receives the history information from theendoscope device 1. The history information is saved on the storage unit73.

The image display device 50 b executes the processing shown in FIG. 29 .The data acquisition unit 52 acquires the history information from thestorage unit 73 instead of the sensor data in Step S120. The structureestimation unit 58 estimates a structure of a subject in a connectionregion by using the history information instead of the sensor data andgenerates structure information in Step S121.

For example, the structure estimation unit 58 estimates the structure ofthe subject by executing the following processing. The first 3D data andthe second 3D data include 3D coordinates in each of a first section, asecond section, and a third section. The first section includes thestart end of a 3D shape. The second section includes the terminal end ofthe 3D shape. The third section is disposed between the first sectionand the second section. For example, the second section in the first 3Ddata and the first section in the second 3D data correspond to aconnection region.

The structure estimation unit 58 analyzes the history information ineach of the second section in the first 3D data, the third section inthe first 3D data close to the second section, the first section in thesecond 3D data, and the third section in the second 3D data close to thefirst section. When the structure estimation unit 58 determines that theinsertion unit 2 does not bend in any of the sections, the structureestimation unit 58 determines that the subject is a straight pipe. Whenthe structure estimation unit 58 determines that the insertion unit 2bends by almost 90 degrees as the distal end 20 of the insertion unit 2passes through the above-described sections, the structure estimationunit 58 determines that the subject is a 90-degree fitting.

Each aspect of the present invention may include the following modifiedexample. The first 3D data are generated by using two or more firstimages. The second 3D data are generated by using two or more secondimages. The two or more first images and the two or more second imagesare generated on the basis of an optical image of a subject acquired bythe insertion unit 2. The insertion unit 2 is inserted inside an objecthaving the subject and is bendable. The structure information isgenerated on the basis of information indicating a bending direction anda bending amount of the insertion unit 2.

Fourth Embodiment

A fourth embodiment of the present invention will be described. In thefourth embodiment, the image display device 50 shown in FIG. 1 is used.Reference data are used as structure information. For example, thereference data are design data including a design value of a geometricstructure. For example, the design data are generated by using 3Dcomputer-aided design (3D-CAD).

Processing executed by the image display device 50 will be described byusing FIG. 30 . FIG. 30 shows a procedure of the processing executed bythe image display device 50. The same processing as that shown in FIG. 2will not be described.

After Step S103, the information acceptance unit 57 accepts thereference data (Step S130). For example, the communication unit 72receives the reference data from an external device. The informationacceptance unit 57 accepts the reference data received by thecommunication unit 72.

After Step S130, the display control unit 56 displays an image of thereference data on the display unit 71 (Step S131).

FIG. 31 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG61 on the display unit71. The image IMG61 includes a region R61, a region R62, and a regionR63. An image of a first 3D shape 3D61 is displayed in the region R61.The first 3D shape 3D61 is indicated by the first 3D data. An image of asecond 3D shape 3D62 is displayed in the region R62. The second 3D shape3D62 is indicated by the second 3D data. An image of a third 3D shape3D63 is displayed in the region R63. The third 3D shape 3D63 isindicated by the reference data.

When the information acceptance unit 57 accepts the reference data,three types of images shown in FIG. 31 are displayed. When theinformation acceptance unit 57 accepts the reference data, the image ofthe reference data does not need to be displayed on the display unit 71.Accordingly, Step S131 does not need to be executed.

After Step S131, the parameter generation unit 53 generates a conversionparameter by using the reference data (Step S132). After Step S132, StepS107 is executed.

The parameter generation unit 53 executes the following processing inStep S132. The parameter generation unit 53 executes shape matching byusing the first 3D data and the reference data. In this way, theparameter generation unit 53 detects a 3D shape of the reference datacorresponding to the 3D shape of the first 3D data. The parametergeneration unit 53 executes shape matching by using the second 3D dataand the reference data. In this way, the parameter generation unit 53detects a 3D shape of the reference data corresponding to the 3D shapeof the second 3D data.

A known method may be applied to 3D shape matching. The parametergeneration unit 53 may execute matching processing by using colorinformation in addition to shape information.

The parameter generation unit 53 generates a position-and-postureconversion parameter used for causing the position and the posture ofthe first 3D coordinate system to match the position and the posture ofthe common coordinate system, respectively.

The parameter generation unit 53 generates a scale conversion parameterused for causing the scale of the first 3D coordinate system to matchthe scale of the common coordinate system. The common coordinate systemis a 3D coordinate system of the reference data.

The parameter generation unit 53 generates a position-and-postureconversion parameter used for causing the position and the posture ofthe second 3D coordinate system to match the position and the posture ofthe common coordinate system, respectively. The parameter generationunit 53 generates a scale conversion parameter used for causing thescale of the second 3D coordinate system to match the scale of thecommon coordinate system.

FIG. 32 shows an example of an image displayed on the display unit 71.The same parts as those shown in FIG. 31 will not be described. Thedisplay control unit 56 displays an image IMG62 on the display unit 71.The image IMG62 includes a region R64. An image of a first 3D shape3D61, a second 3D shape 3D62, and a third 3D shape 3D63 is displayed inthe region R64. A user can confirm that the first 3D data and the second3D data are disposed at a correct position in the reference data. Inthis way, the user can confirm that the first 3D data and the second 3Ddata are accurately connected to each other.

Reference data used as structure information are not limited to designdata. The reference data may be data other than the design data.

For example, 3D data generated by using two or more images of aninspection target generated in a previous inspection may be used asreference data. The 3D data are generated in the processing shown inFIG. 27 . The 3D data are different from any of the first 3D data andthe second 3D data. The 3D data used as the reference data may begenerated by equipment other than endoscope equipment that generates 3Ddata of a subject inside an inspection target.

As long as the parameter generation unit 53 can cause the positions ofthe first 3D data and the second 3D data to match the position ofreference data, the reference data are not necessarily 3D data. Thereference data may be two-dimensional data (2D data). The parametergeneration unit 53 converts the 3D data into 2D information in order tocause the position of the 3D data and the position of the 2D data tomatch each other. The 2D information indicates a two-dimensional shapein an arbitrary plane. The parameter generation unit 53 causes theposition of the 2D information and the position of the 2D data to matcheach other.

Each aspect of the present invention may include the following modifiedexample. The structure information is configured as design dataincluding a design value of a geometric structure of a subject or isconfigured as 3D data different from any of the first 3D data and thesecond 3D data.

In the fourth embodiment, the image display device 50 can connect thefirst 3D data and the second 3D data together and can display an imageof a 3D shape of a wide range on an image of the reference data.

First Modified Example of Fourth Embodiment

A first modified example of the fourth embodiment of the presentinvention will be described. In the fourth embodiment described above,the ID of each blade is not used for connecting the first 3D data andthe second 3D data together. On the other hand, in the first modifiedexample of the fourth embodiment, the first 3D data and the second 3Ddata are connected together such that the ID of each blade of the first3D data and the second 3D data matches the ID of each blade of thereference data.

In the first modified example of the fourth embodiment, the imagedisplay device 50 shown in FIG. 1 is used. The storage unit 73 storesreference data of a gas turbine. FIG. 33 shows reference data of a gasturbine GT70. In the example shown in FIG. 33 , the gas turbine GT70includes twelve blades. An ID is allocated to each of the blades. Thereference data shown in FIG. 33 may be a cross-section of the gasturbine GT70 or may be 3D-CAD data.

Each ID corresponds to a position of a blade. For example, in a case inwhich sequential IDs are allocated to the blades, two sequential IDs areallocated to two blades adjacent to each other. Structure information inthe first modified example of the fourth embodiment is the referencedata and the ID allocated to each blade.

The parameter generation unit 53 executes processing shown in FIG. 34 inStep S106. FIG. 34 shows a procedure of the processing executed by theparameter generation unit 53.

The parameter generation unit 53 identifies blade regions in the first3D data (Step S106 j). Each of the blade regions includes one blade.

The parameter generation unit 53 may use any method in order to identifya blade region. For example, the parameter generation unit 53 extractstwo or more similar shapes in the first 3D data as blade regions.Alternatively, the parameter generation unit 53 executes shape matchingby using a shape of a blade registered in advance and the first 3D dataand extracts a blade region including a shape that matches the shape ofthe blade.

After Step S106 j, the parameter generation unit 53 allocates IDs to theblade regions identified in Step S106 j (Step S106 k).

The parameter generation unit 53 may use any method in order to allocateIDs to the blade regions. For example, an image of the first 3D data isdisplayed on the display unit 71. A user inputs an ID of each blade intothe image display device 50 by operating the operation unit 70. Theparameter generation unit 53 allocates the ID to the blade region. Theparameter generation unit 53 may detect a reference blade having adistinctive structure from the first 3D data and may alternatelyallocate an ID to each blade starting from the reference blade.

When endoscope equipment generates an image used for generating 3D data,a turning tool may be used. The turning tool automatically rotates adisk. The turning tool outputs information of a rotation angle of thedisk to the endoscope equipment. The endoscope equipment determines aposition of a blade seen in an image on the basis of the information.

For example, before the turning tool rotates the disk, the endoscopeequipment sets a blade seen in an image as a reference blade. The totalnumber of blades are known, and the angle between two adjacent blades isknown. While the disk rotates, the endoscope equipment determines ablade seen in an image on the basis of the rotation angle of the disk.When the disk rotates once, the endoscope equipment can determine thatthe reference blade is seen in an image again.

The endoscope equipment allocates an ID to each blade and embeds the IDin an image. The endoscope equipment may allocate an ID to each bladewithout using the turning tool and may embed the ID in an image.

3D coordinates of each point included in 3D data are associated with animage used for generating the 3D data. The parameter generation unit 53acquires IDs of images associated with the blade regions identified inStep S106 j and allocates the IDs to the blade regions.

After Step S106 k, the parameter generation unit 53 identifies bladeregions in the second 3D data (Step S1061). Step S1061 is similar toStep S106 j.

After Step S1061, the parameter generation unit 53 allocates IDs to theblade regions identified in Step S1061 (Step S106 m). Step S106 m issimilar to Step S106 k.

After Step S106 m, the parameter generation unit 53 calculates aconversion parameter used for correcting the position, the posture, andthe scale of the first 3D data such that the ID of a blade in thereference data and the ID of a blade region in the first 3D data matcheach other (Step S106 n).

After Step S106 n, the parameter generation unit 53 calculates aconversion parameter used for correcting the position, the posture, andthe scale of the second 3D data such that the ID of a blade in thereference data and the ID of a blade region in the second 3D data matcheach other (Step S106 o). When Step S106 o is executed, the processingshown in FIG. 34 is completed.

FIG. 35 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG71 on the display unit71. The image IMG71 includes a region R71. An image of a first 3D shape3D71, a second 3D shape 3D72, and a third 3D shape 3D73 is displayed inthe region R71. The first 3D shape 3D71 is indicated by the first 3Ddata. The second 3D shape 3D72 is indicated by the second 3D data. Thethird 3D shape 3D73 is indicated by the reference data.

The first 3D shape 3D71 includes two blades. The IDs of the two bladesare 2 and 3. The position of the first 3D data is set to thatoverlapping two blades of the reference data to which 2 and 3 areallocated as IDs.

The second 3D shape 3D72 includes five blades. The IDs of the fiveblades are 7 to 11. The position of the second 3D data is set to thatoverlapping five blades of the reference data to which 7 to 11 areallocated as IDs.

There is no blade to which 1, 4, 5, 6, or 12 is allocated as an ID inthe first 3D data or the second 3D data. The blades of the referencedata to which these IDs are allocated are displayed.

The display control unit 56 may display each blade on the display unit71 so that a user can distinguish a blade included in the first 3D dataor the second 3D data and a blade included in the reference data fromeach other. For example, the color of a blade included in the first 3Ddata or the second 3D data may be different from that of a bladeincluded in the reference data. The user can check a blade included inthe 3D data and a blade not included in the 3D data.

The parameter generation unit 53 may execute the processing shown inFIG. 19 in addition to the processing shown in FIG. 34 . In this way,the parameter generation unit 53 can more accurately connect the first3D data and the second 3D data together.

Each aspect of the present invention may include the following modifiedexample. A subject includes two or more objects. The structureinformation indicates positions at which the two or more objects aredisposed.

In the first modified example of the fourth embodiment, the parametergeneration unit 53 generates a conversion parameter by using an ID asstructure information. Therefore, the image display device 50 canaccurately connect the first 3D data and the second 3D data together.

Second Modified Example of Fourth Embodiment

A second modified example of the fourth embodiment of the presentinvention will be described. In the second modified example of thefourth embodiment, the image display device 50 shown in FIG. 1 is used.

A subject in the first modified example of the fourth embodimentdescribed above is a blade of a gas turbine. On the other hand, asubject in the second modified example of the fourth embodiment is acombustion chamber of a gas turbine. The combustion chamber includesfuel injection nozzles and plates as typical inspection portions. Theplates are disposed around the fuel injection nozzles.

The fuel injection nozzles are circularly disposed like blades in manycases. An ID is allocated to each fuel injection nozzle in many cases.Each ID has a value in accordance with a position at which each fuelinjection nozzle is disposed. Each fuel injection nozzle is disposed onan arbitrary periphery of a circle, and a center position of the circleis defined.

FIG. 36 shows a structure of a combustion chamber CC80. The combustionchamber CC80 includes twelve fuel injection nozzles NZ80. The actualnumber of fuel injection nozzles is different in accordance with theengine type. The twelve fuel injection nozzles NZ80 are disposed on thefront side of the combustion chamber CC80. A center position CP80indicates the center of the circle on which the twelve fuel injectionnozzles NZ80 are disposed. In the second modified example of the fourthembodiment, the first 3D data and the second 3D data are connectedtogether by using IDs allocated to the fuel injection nozzles.

The information INF10 of the dialog box DL10 shown in FIG. 4 includesinformation of a gas turbine, which is an inspection target, andinformation of a combustion chamber in the gas turbine. For example, auser selects a “gas turbine” and then selects a “combustion chamber.”When the user inputs information indicating that the selection isfinalized into the image display device 50, the information acceptanceunit 57 accepts the information selected by the user. At this time, theinformation acceptance unit 57 accepts a character string “combustionchamber.”

For example, the structure information of the combustion chamberindicates that fuel injection nozzles in the first 3D data and thesecond 3D data are disposed at regular intervals on a periphery of acircle having an arbitrary curvature radius. The structure informationindicates that the sizes of the fuel injection nozzles are the same. Theparameter generation unit 53 generates a conversion parameter on thebasis of the structure information in Step S106.

The total number of fuel injection nozzles, the size of each fuelinjection nozzle, and the ID of each fuel injection nozzle are known ina case in which the engine type has already been identified. Thestructure information may include the total number of fuel injectionnozzles, the size of each fuel injection nozzle, and the ID of each fuelinjection nozzle. In this case, the parameter generation unit 53 cangenerate a position-and-posture conversion parameter used for accuratelyconnecting the first 3D data and the second 3D data together.

The parameter generation unit 53 executes the processing shown in FIG.34 in Step S106. The processing shown in FIG. 34 is also applied to acase in which a subject is a combustion chamber.

FIG. 37 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG81 on the display unit71. The image IMG81 includes a region R81. An image of a first 3D shape3D81, a second 3D shape 3D82, and a third 3D shape 3D83 is displayed inthe region R81. The first 3D shape 3D81 is indicated by the first 3Ddata. The second 3D shape 3D82 is indicated by the second 3D data. Thethird 3D shape 3D83 is indicated by the reference data.

The first 3D shape 3D81 includes four fuel injection nozzles. The IDs ofthe four fuel injection nozzles are 2 to 5. The position of the first 3Ddata is set to that overlapping four fuel injection nozzles of thereference data to which 2 to 5 are allocated as IDs.

The second 3D shape 3D82 includes four fuel injection nozzles. The IDsof the four fuel injection nozzles are 9 to 12. The position of thesecond 3D data is set to that overlapping four fuel injection nozzles ofthe reference data to which 9 to 12 are allocated as IDs.

There is no fuel injection nozzle to which 1, 6, 7, or 8 is allocated asan ID in the first 3D data or the second 3D data. Fuel injection nozzlesof the reference data to which these IDs are allocated are displayed.

The display control unit 56 may display each fuel injection nozzle onthe display unit 71 so that a user can distinguish a fuel injectionnozzle included in the first 3D data or the second 3D data and a fuelinjection nozzle included in the reference data from each other. Forexample, the color of a fuel injection nozzle included in the first 3Ddata or the second 3D data may be different from that of a fuelinjection nozzle included in the reference data. A user can check a fuelinjection nozzle included in the 3D data and a fuel injection nozzle notincluded in the 3D data.

In the second modified example of the fourth embodiment, the parametergeneration unit 53 generates a conversion parameter by using an ID asstructure information. Therefore, the image display device 50 canaccurately connect the first 3D data and the second 3D data together.

The first and second modified examples of the fourth embodiment areapplied to an inspection of a gas turbine of an aircraft engine. Thesemodified examples may be applied to an inspection of a tube-likestructure. For example, these modified examples may be applied to aninspection of a heat exchange tube. The heat exchange tube includes tensof thin tubes. An ID may be allocated to each tube. The parametergeneration unit 53 can connect the first 3D data and the second 3D datatogether by using the ID.

Fifth Embodiment

A fifth embodiment of the present invention will be described. In thefifth embodiment, the structure information is configured as cameratrace data. The camera trace data indicate two or more positions atwhich a camera that generates two or more images used for generating 3Ddata is disposed. Each of the positions corresponds to the cameracoordinate described above. In other words, the camera trace dataindicate traces of the camera. The camera trace data are associated withthe first 3D data and the second 3D data. In a case in which theendoscope device 1 shown in FIG. 22 is used, the camera trace dataindicate two or more positions at which the distal end 20 of theinsertion unit 2 is disposed. The first 3D data and the second 3D dataare connected together by using the camera trace data.

The image display device 50 shown in FIG. 1 is changed to an imagedisplay device 50 c shown in FIG. 38 . FIG. 38 shows a configuration ofthe image display device 50 c. The same configuration as that shown inFIG. 1 will not be described.

The image display device 50 c shown in FIG. 38 includes a control unit51, a data acquisition unit 52, a parameter generation unit 53, aconversion unit 54, a data generation unit 55, a display control unit56, and a lost section calculation unit 60. The image display device 50c does not include the information acceptance unit 57 and the structureestimation unit 58 shown in FIG. 1 .

Each unit of the image display device 50 c may be constituted by atleast one of a processor and a logic circuit. Each unit of the imagedisplay device 50 c may include one or a plurality of processors. Eachunit of the image display device 50 c may include one or a plurality oflogic circuits.

The storage unit 73 stores first camera trace data and second cameratrace data. The first camera trace data are associated with the first 3Ddata, and the second camera trace data are associated with the second 3Ddata. The data acquisition unit 52 connects to the storage unit 73 andacquires the first camera trace data and the second camera trace datafrom the storage unit 73.

The lost section calculation unit 60 calculates a position of a lostsection by using the first camera trace data and the second camera tracedata. The lost section includes 3D data corresponding to a lost regionof a subject. The lost region is not included in a first region or asecond region of the subject. The first region corresponds to the 3Dcoordinates included in the first 3D data. The second region correspondsto the 3D coordinates included in the second 3D data. The lost sectioncalculation unit 60 can calculate the length of the lost section on thebasis of the position of the lost section.

Processing executed by the image display device 50 c will be describedby using FIG. 39 . FIG. 39 shows a procedure of the processing executedby the image display device 50 c. The same processing as that shown inFIG. 2 will not be described.

The data acquisition unit 52 connects to the storage unit 73 andacquires the first 3D data and the first camera trace data from thestorage unit 73 (Step S140). After Step S140, the display control unit56 displays an image of the first 3D data and the first camera tracedata on the display unit 71 (Step S141).

After Step S141, the data acquisition unit 52 connects to the storageunit 73 and acquires the second 3D data and the second camera trace datafrom the storage unit 73 (Step S142). After Step S142, the displaycontrol unit 56 displays an image of the second 3D data and the secondcamera trace data on the display unit 71 (Step S143).

The order of Steps S140 to S143 is not limited to that shown in FIG. 39. For example, Step S140 and Step S141 may be executed after Step S142and Step S143 are executed. Alternatively, Step S141 and Step S143 maybe executed after Step S140 and

Step S142 are executed. Step S141 and Step S143 may be omitted.

After Step S143, the parameter generation unit 53 generates a conversionparameter on the basis of the relationship between the first cameratrace data and the second camera trace data (Step S144). The parametergeneration unit 53 generates a position-and-posture conversion parameterused for connecting the first 3D data and the second 3D data together byusing the following method. The following method can be applied to aninspection of a pipe and an inspection of a blade (rotor blade).

In a case in which a subject is a pipe, the parameter generation unit 53executes the following processing. The parameter generation unit 53calculates a first straight line that approximates the traces of thecamera indicated by the first camera trace data. In addition, theparameter generation unit 53 calculates a second straight line thatapproximates the traces of the camera indicated by the second cameratrace data. The parameter generation unit 53 generates aposition-and-posture conversion parameter used for correcting thepositions and the postures of the first 3D data and the second 3D datasuch that the first straight line and the second straight line matcheach other.

The parameter generation unit 53 may calculate a first straight line byusing the first camera trace data associated with the entire first 3Ddata. The parameter generation unit 53 may calculate a first straightline by using the first camera trace data associated with a connectionregion identified on the basis of the timestamps of the first 3D data.

The parameter generation unit 53 may calculate a second straight line byusing the second camera trace data associated with the entire second 3Ddata. The parameter generation unit 53 may calculate a second straightline by using the second camera trace data associated with a connectionregion identified on the basis of the timestamps of the second 3D data.

FIG. 40 shows an example of an image displayed on the display unit 71before the first 3D data and the second 3D data are connected together.The display control unit 56 displays an image IMG91 on the display unit71. The image IMG91 includes a region R91 and a region R92. An image ofa first 3D shape 3D91 is displayed in the region R91. The first 3D shape3D91 is indicated by the first 3D data. An image of a second 3D shape3D92 is displayed in the region R92. The second 3D shape 3D92 isindicated by the second 3D data.

First camera traces TR91 and second camera traces TR92 are shown in FIG.40 . The first camera traces TR91 indicate two or more positionsincluded in the first camera trace data. The second camera traces TR92indicate two or more positions included in the second camera trace data.The display control unit 56 does not need to display the first cameratraces TR91 and the second camera traces TR92 on the display unit 71.

FIG. 41 shows an example of an image displayed on the display unit 71after the first 3D data and the second 3D data are connected together.In FIG. 41 , Step S145 described later is not considered. The same partsas those shown in FIG. 40 will not be described. The display controlunit 56 displays an image IMG92 on the display unit 71. The image IMG92includes a region R93. An image of a first 3D shape 3D91 and a second 3Dshape 3D92 is displayed in the region R93.

The first 3D shape 3D91 and the second 3D shape 3D92 are disposed suchthat a first straight line approximating first camera traces TR91 and asecond straight line approximating second camera traces TR92 match eachother. A line L91 is constituted by the first straight line and thesecond straight line.

In a case in which a subject is a blade (rotor blade), the parametergeneration unit 53 executes the following processing. The parametergeneration unit 53 calculates a first curved line that approximates thetraces of the camera indicated by the first camera trace data. Inaddition, the parameter generation unit 53 calculates a second curvedline that approximates the traces of the camera indicated by the secondcamera trace data. Each of the first curved line and the second curvedline has a circular arc shape. The parameter generation unit 53generates a position-and-posture conversion parameter used forcorrecting the positions and the postures of the first 3D data and thesecond 3D data such that the first curved line and the second curvedline are connected together on a periphery of a circle having anarbitrary diameter.

FIG. 42 shows an example of an image displayed on the display unit 71before the first 3D data and the second 3D data are connected together.The display control unit 56 displays an image IMG101 on the display unit71. The image IMG101 includes a region R101 and a region R102. An imageof a first 3D shape 3D101 is displayed in the region R101. The first 3Dshape 3D101 is indicated by the first 3D data. An image of a second 3Dshape 3D102 is displayed in the region R102. The second 3D shape 3D102is indicated by the second 3D data.

First camera traces TR101 and second camera traces TR102 are shown inFIG. 42 . The first camera traces TR101 indicate two or more positionsincluded in the first camera trace data. The second camera traces TR102indicate two or more positions included in the second camera trace data.The display control unit 56 does not need to display the first cameratraces TR101 and the second camera traces TR102 on the display unit 71.

FIG. 43 shows an example of an image displayed on the display unit 71after the first 3D data and the second 3D data are connected together.In FM. 43, Step S145 described later is not considered. The same partsas those shown in FIG. 42 will not be described. The display controlunit 56 displays an image IMG102 on the display unit 71. The imageIMG102 includes a region R103. An image of a first 3D shape 3D101 and asecond 3D shape 3D102 is displayed in the region R103.

The first 3D shape 3D101 and the second 3D shape 3D102 are disposed suchthat a first curved line approximating first camera traces TR101 and asecond curved line approximating second camera traces TR102 are disposedon the same periphery of a circle. A circle CR101 is constituted by thefirst curved line and the second curved line.

The parameter generation unit 53 can generate a position-and-postureconversion parameter in accordance with a subject by using theabove-described method. In a case in which the subject is a pipe, thescales of the first 3D data and the second 3D data are not corrected. Ina case in which the subject is a blade (rotor blade), the scales of thefirst 3D data and the second 3D data may be corrected.

After Step S144, the lost section calculation unit 60 calculates a lostsection in a connection region by using the first camera trace data andthe second camera trace data (Step S145). The lost section calculationunit 60 calculates a position of the lost section by using the followingmethod. The following method can be applied to an inspection of a pipeand an inspection of a blade (rotor blade).

In a case in which a subject is a pipe, the lost section calculationunit 60 executes the following processing. FIG. 44 shows a method ofcalculating a lost section.

Each of two or more points P111, two or more points P112, and two ormore points P113 indicates a position (3D coordinates) of the camerawhen the camera generates an image. The two or more points P111constitute first camera trace data and are associated with first 3D data3D111. The two or more points P112 constitute second camera trace dataand are associated with second 3D data 3D112. The two or more pointsP113 indicate positions of the camera in a region for which there are no3D data.

Since there are no 3D data corresponding to the two or more points P113,there are no camera trace data corresponding to the two or more pointsP113. The two or more points P113 constitute a lost section.

The first 3D data 3D111 are generated by using two or more images FR111.The second 3D data 3D112 are generated by using two or more imagesFR112. Two or more images FR113 are not used for generating the first 3Ddata 3D111 or the second 3D data 3D112. The two or more images FR111,the two or more images FR112, and the two or more images FR113 areincluded in one video file. Each of the two or more images FR111, thetwo or more images FR112, and the two or more images FR113 has a framenumber.

Each of the two or more points P111 is associated with the image FR111.Each of the two or more points P112 is associated with the image FR112.None of the two or more points P113 is associated with the image FR113.

All the two or more images FR111 are not necessarily used for generatingthe first 3D data 3D111. Two or more key frames included in the two ormore images FR111 are used for generating the first 3D data 3D111. Thenumber of the key frames is less than or equal to the total number ofthe two or more images FR111. Similarly, two or more key frames includedin the two or more images FR112 are used for generating the second 3Ddata 3D112.

The number of the points P111 and the number of the images FR111 (keyframes) have correlation with each other. The number of the points P112and the number of the images FR112 (key frames) have correlation witheach other. The lost section calculation unit 60 calculates the numberof the points P111 and the points P112 per one image. The lost sectioncalculation unit 60 calculates the number of the points P113 by usingboth the number of the points P111 and the points P112 per one image andthe number of the images FR113.

The lost section calculation unit 60 calculates an interval(three-dimensional distance) between the points P111 or the points P112.The lost section calculation unit 60 calculates a length LT111 of thelost section by using both the interval and the number of the pointsP113. The above-described method is not used in a case in which videodata used for generating the first 3D data are different from that usedfor generating the second 3D data.

In a case in which a subject is a blade (rotor blade), the last sectioncalculation unit 60 executes the following processing. FIG. 45 and FIG.46 show a method of calculating a lost section.

FIG. 45 shows a relationship between camera trace data and 3D data. Eachof two or more points P121, two or more points P122, two or more pointsP123, and two or more points P124 indicates a position (3D coordinates)of the camera when the camera generates an image. The two or more pointsP121 constitute first camera trace data and are associated with first 3Ddata 3D121. The two or more points P122 constitute second camera tracedata and are associated with second 3D data 3D122. The two or morepoints P123 and the two or more points P124 indicate positions of thecamera in a region for which there are no 3D data.

Since there are no 3D data corresponding to the two or more points P123,there are no camera trace data corresponding to the two or more pointsP123. Since there are no 3D data corresponding to the two or more pointsP124, there are no camera trace data corresponding to the two or morepoints P124. The two or more points P123 constitute a first lostsection. The two or more points P124 constitute a second lost section.

FIG. 46 shows a video file. First 3D data 3D121 are generated by usingtwo or more images FR121. Second 3D data 3D122 are generated by usingtwo or more images FR122. Two or more images FR123 and two or moreimages FR124 are not used for generating the first 3D data 3D121 or thesecond3D data 3D122. The two or more images FR121, the two or moreimages FR122, the two or more images FR123, and the two or more imagesFR124 are included in one video file. Each of the two or more imagesFR121, the two or more images FR122, the two or more images FR123, andthe two or more images FR124 has a frame number.

Each of the two or more points P121 is associated with the image FR121.Each of the two or more points P122 is associated with the image FR122.None of the two or more points P123 is associated with the image FR123.None of the two or more points P124 is associated with the image FR124.

All the two or more images FR121 are not necessarily used for generatingthe first 3D data 3D121. Two or more key frames included in the two ormore images FR121 are used for generating the first 3D data 3D121. Thenumber of the key frames is less than or equal to the total number ofthe two or more images FR121. Similarly, two or more key frames includedin the two or more images FR122 are used for generating the second 3Ddata 3D122.

The number of the points P121 and the number of the images FR121 (keyframes) have correlation with each other. The number of the points P122and the number of the images FR122 (key frames) have correlation witheach other. The lost section calculation unit 60 calculates the numberof the points P121 and the points P122 per one image. The lost sectioncalculation unit 60 calculates the number of the points P123 by usingboth the number of the points P121 and the points P122 per one image andthe number of the images FR123.

The lost section calculation unit 60 calculates an interval (angle)between the points P121 or the points P122. The lost section calculationunit 60 calculates an angle AG121 of the first lost section by usingboth the interval and the number of the points P123. The lost sectioncalculation unit 60 may calculate an interval (angle) between blades byusing both the interval between the points P121 and the number of bladesin the first 3D data 3D121. The lost section calculation unit 60 maycalculate the number of blades in the first lost section by using boththe interval and the angle AG121 of the first lost section.

The lost section calculation unit 60 subtracts a first angle, a secondangle, and the angle AG121 from the angle (360 degrees) around a blade,thus calculating an angle AG122 of the second lost section. The firstangle corresponds to a range in which blades of the first 3D data 3D121are disposed. The second angle corresponds to a range in which blades ofthe second 3D data 3D122 are disposed. The lost section calculation unit60 may subtract the number of blades of the first 3D data 3D121, thenumber of blades of the second 3D data 3D122, and the number of bladesin the first lost section from the total number of blades, thuscalculating the number of blades in the second lost section. Theabove-described method is not used in a case in which video data usedfor generating the first 3D data are different from that used forgenerating the second 3D data.

After Step S145, the parameter generation unit 53 generates a conversionparameter on the basis of the lost section in Step S106. At this time,the parameter generation unit 53 generates a conversion parameter usedfor separating the first 3D data and the second 3D data from each otherby the length or the angle of the lost section.

Each aspect of the present invention may include the following modifiedexample. The structure information indicates two or more positions atwhich the distal end 20 of the insertion unit 2 to be inserted inside anobject having a subject is disposed.

Each aspect of the present invention may include the following modifiedexample. The structure information includes first position information(first camera trace data) and second position information (second cameratrace data). The first position information indicates two or morepositions at which the distal end 20 is disposed in order to acquire twoor more first images. The second position information indicates two ormore positions at which the distal end 20 is disposed in order toacquire two or more second images. The parameter generation unit 53generates a position conversion parameter and a posture conversionparameter used for converting the first 3D coordinate system and thesecond 3D coordinate system into a common coordinate system on the basisof the first position information and the second position information ina generation step (Step S144). The conversion unit 54 converts the first3D coordinate system and the second 3D coordinate system into the commoncoordinate system by using the position conversion parameter and theposture conversion parameter in a conversion step (Step S107).

Each aspect of the present invention may include the following modifiedexample. The first 3D data are generated by using the two or more firstimages. The second 3D data are generated by using the two or more secondimages. The two or more first images and the two or more second imagesare included in one video file.

Each aspect of the present invention may include the following modifiedexample. The lost section calculation unit 60 calculates a position of alost region on the basis of the number of the two or more first images,the number of the two or more second images, and the number of thirdimages in a calculation step (Step S145). The third images aretemporally disposed between the two or more first images and the two ormore second images in the video file. The lost region is a region of asubject different from any one of a first region of the subject and asecond region of the subject. The first region corresponds to the 3Dcoordinates included in the first 3D data. The second region correspondsto the 3D coordinates included in the second 3D data. The conversionunit 54 converts the first 3D coordinate system and the second 3Dcoordinate system into the common coordinate system on the basis of theposition of the lost region in the conversion step (Step S107).

In the fifth embodiment, the image display device 50 c can connect thefirst 3D data and the second 3D data together and can display an imageof a 3D shape of a wide range. In addition, the image display device 50c can display an image of a 3D shape close to an original shape of asubject by considering the lost section.

Modified Example of Fifth Embodiment

A modified example of the fifth embodiment of the present invention willbe described. In the fifth embodiment described above, camera trace datagenerated through image processing are used. On the other hand, in themodified example of the fifth embodiment, a position of a camera iscalculated by using sensor data, and data indicating the position areused instead of the camera trace data. In the modified example of thefifth embodiment, the image display device 50 c shown in FIG. 38 isused.

For example, an IMU or an insertion-length sensor is used. The IMU isdisposed in the distal end 20 of the insertion unit 2. Theinsertion-length sensor is disposed in the base end portion of theinsertion unit 2 shown in FIG. 22 or is disposed in a drum that housesthe insertion unit 2. The insertion-length sensor determines a length(insertion-length) of a part of the insertion unit 2 inserted into aninspection target. Processing executed by the image display device 50 cwill be described by using FIG. 47 . FIG. 47 shows a procedure of theprocessing executed by the image display device 50 c. The sameprocessing as that shown in FIG. 2 will not be described.

After Step S103, the data acquisition unit 52 connects to the storageunit 73 and acquires sensor data in a connection region from the storageunit 73 (Step S120). For example, the data acquisition unit 52 acquiressensor data in a section from the time point of the terminal end of thefirst 3D data to the time point of the start end of the second 3D datafrom the storage unit 73. Step S120 is the same as that shown in FIG. 29.

After Step S120, the parameter generation unit 53 generates a conversionparameter on the basis of the sensor data acquired in Step S120 (StepS150).

For example, the parameter generation unit 53 calculates traces of thedistal end 20 of the insertion unit 2 by using sensor data of an IMU.The traces indicate two or more positions at which the distal end 20 isdisposed. The parameter generation unit 53 determines movement of thedistal end 20 on the basis of the traces. Specifically, the parametergeneration unit 53 determines whether the distal end 20 moves linearlyor bends by 90 degrees. In addition, the lost section calculation unit60 calculates a length of a lost section by using the method in thefifth embodiment.

In a case in which sensor data of an insertion-length sensor are used,the parameter generation unit 53 cannot determine whether the distal end20 moves linearly or bends by 90 degrees. The lost section calculationunit 60 can calculate the length of the lost section by using the sensordata of the insertion-length sensor. In a case in which the historyinformation of operations related to bending of the insertion unit 2 inthe second modified example of the third embodiment is used, theparameter generation unit 53 can determine whether the distal end 20moves linearly or bends by 90 degrees on the basis of the historyinformation. The lost section calculation unit 60 can calculate a shapeof the lost section on the basis of the history information.

When stator blades of a gas turbine are inspected, the distal end of ascope moves above the periphery of the disk of the stator blades. Evenin this case, the parameter generation unit 53 can calculate the lengthof the lost section by using the sensor data of the insertion-lengthsensor.

FIG. 48 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG131 on the display unit71. The image IMG131 includes a region R131. An image of a first 3Dshape 3D131 and a second 3D shape 3D132 is displayed in the region R131.The first 3D shape 3D131 is indicated by the first 3D data. The second3D shape 3D132 is indicated by the second 3D data.

In the example shown in FIG. 48 , the parameter generation unit 53determines that the distal end 20 bends by 90 degrees. Therefore, theparameter generation unit 53 generates a position-and-posture conversionparameter used for causing the center axis of the second 3D shape 3D132to be orthogonal to the center axis of the first 3D shape 3D131. Thefirst 3D shape 3D131 and the second 3D shape 3D132 are disposed suchthat an angle AG131 between the first 3D shape 3D131 and the second 3Dshape 3D132 becomes 90 degrees.

In the second modified example of the third embodiment described above,the parameter generation unit 53 generates a conversion parameter byusing the structure information. On the other hand, in the modifiedexample of the fifth embodiment, the parameter generation unit 53generates a conversion parameter by using the sensor data without usingthe structure information.

Each aspect of the present invention may include the following modifiedexample. The lost section calculation unit 60 calculates a shape of thelost region in a calculation step (Step S150). The conversion unit 54converts the first 3D coordinate system and the second 3D coordinatesystem into a common coordinate system on the basis of the shape of thelost region in a conversion step (Step S107).

In the modified example of the fifth embodiment, the image displaydevice 50 c can connect the first 3D data and the second 3D datatogether and can display an image of a 3D shape of a wide range. Inaddition, the image display device 50 c can display an image of a 3Dshape by considering the lost section.

Sixth Embodiment

A sixth embodiment of the present invention will be described.Industrial endoscope devices have been used for an inspection ofinternal abnormalities (damage, corrosion, and the like) of boilers, gasturbines, automobile engines, pipes, and the like. In order to enablevalidation of contents of an inspection performed by an inspector, avideo is generally recorded during the inspection.

A technique of reconfiguring a 3D shape of an inspection target by usinga video is known. The video is generated on the basis of an opticalimage acquired through a single-eye optical system. By using thistechnique, an endoscope device can acquire 3D data of a wide range ofthe inspection target.

3D data of an inspection target may be used so that a user can recognizea region that has been inspected and a region that has not beeninspected. 3D data of the region that has been inspected are generated,but 3D data of the region that has not been inspected are not generated.In a case in which the inspection target has a unique shape, the usercan recognize which region of the inspection target actually correspondsto a region of the 3D data. The user can recognize the inspected regionin the inspection target by checking the 3D shape indicated by the 3Ddata.

However, a heat exchange tube, a gas turbine, or the like, which is atypical inspection target in an endoscopic inspection, has only a fewunique structures. Therefore, it is difficult for a user to recognizewhich region of the inspection target corresponds to a region of the 3Ddata. Even in a case in which a user visually compares the 3D data withreference data of the inspection target, it is difficult for a user torecognize which region of the reference data corresponds to a region ofthe 3D data.

The sixth embodiment provides a method causing a user to recognize aregion that has been inspected and a region that has not been inspectedeven when a subject does not have a unique structure. For example, thesubject includes a structure (for example, a pipe) in which the sameshape continues. Alternatively, the subject includes a plurality ofstructures (for example, blades) having the same shape.

The sixth embodiment is almost the same as the fourth embodiment.However, 3D data are not divided into two pieces in the sixthembodiment. Hereinafter, an example in which a subject is a blade willbe described. The 3D shape indicated by the 3D data is close to acircle.

In the sixth embodiment, the image display device 50 shown in FIG. 1 isused. Processing executed by the image display device 50 will bedescribed by using FIG. 49 . FIG. 49 shows a procedure of the processingexecuted by the image display device 50.

The data acquisition unit 52 connects to the storage unit 73 andacquires the 3D data from the storage unit 73 (Step S200). After StepS200, the display control unit 56 displays an image of the 3D data onthe display unit 71 (Step S201). In the first to fifth embodimentsdescribed above, two types of 3D data are acquired from the storage unit73, and an image of each piece of the 3D data is displayed on thedisplay unit 71. On the other hand, in the sixth embodiment, one type of3D data are acquired from the storage unit 73, and an image of the 3Ddata is displayed on the display unit 71.

After Step S201, the information acceptance unit 57 accepts thereference data (Step S202). For example, the communication unit 72receives the reference data from an external device. The informationacceptance unit 57 accepts the reference data received by thecommunication unit 72. The reference data include structure information.An ID is allocated to each of two or more blades having the same shape.The reference data include the ID of each blade.

The reference data are data generated by using 3D-CAD, data indicating atwo-dimensional shape in an arbitrary plane, or the like. As long as thereference data include the structure information and IDs are associatedwith the reference data, the format of the reference data is not limitedto the above-described examples.

IDs do not need to be allocated to blades before the informationacceptance unit 57 accepts the reference data. A user may allocate theIDs to the blades after the information acceptance unit 57 accepts thereference data. Alternatively, the IDs may be automatically allocated tothe blades.

After Step S202, the display control unit 56 displays an image of thereference data accepted in Step S201 on the display unit 71 (Step S203).

The order of Steps S200 to S203 is not limited to that shown in FIG. 49. For example, Step S200 and Step S201 may be executed after Step S202and Step S203 are executed. Alternatively, Step S201 and Step S203 maybe executed after Step S200 and

Step S202 are executed. Step S201 and Step S203 may be omitted.

After Step S203, the parameter generation unit 53 generates a conversionparameter used for converting the 3D coordinate system of the 3D dataand the 3D coordinate system of the reference data into a commoncoordinate system (Step S204).

Hereinafter, an example in which the 3D coordinate system of thereference data is the reference of coordinate systems and is used as thecommon coordinate system will be described. In the following example,the parameter generation unit 53 generates a position-and-postureconversion parameter used for causing the position and the posture ofthe 3D coordinate system of the 3D data to match the position and theposture of the 3D coordinate system of the reference data, respectively.In addition, the parameter generation unit 53 generates a scaleconversion parameter used for causing the scale of the 3D coordinatesystem of the 3D data to match the scale of the 3D coordinate system ofthe reference data. However, a method of generating a conversionparameter is not limited to the following example. The 3D coordinatesystem of the 3D data may be used as a common coordinate system. Adifferent 3D coordinate system from any of the 3D coordinate system ofthe 3D data and the 3D coordinate system of the reference data may beused as a common coordinate system.

The parameter generation unit 53 executes processing shown in FIG. 50 inStep S204. FIG. 50 shows a procedure of the processing executed by theparameter generation unit 53.

The parameter generation unit 53 identifies blade regions in the 3D data(Step S204 a). Each of the blade regions includes one blade.

After Step S204 a, the parameter generation unit 53 allocates IDs to theblade regions identified in Step S204 a (Step S204 b).

After Step S204 b, the parameter generation unit 53 calculates aconversion parameter used for correcting the position, the posture, andthe scale of the 3D data such that the ID of a blade in the referencedata and the ID of a blade region in the 3D data match each other (StepS204 c). When Step S204 c is executed, the processing shown in FIG. 50is completed.

FIG. 51 shows an example of an image displayed on the display unit 71.The display control unit 56 displays an image IMG141 on the display unit71. The image IMG141 includes a region R141 and a region R142. An imageof a first 3D shape 3D141 is displayed in the region R141. The first 3Dshape 3D141 is indicated by the 3D data. An image of a second 3D shape3D142 is displayed in the region R142. The second 3D shape 3D142 isindicated by the reference data.

The first 3D shape 3D141 includes eleven blades. The IDs of the elevenblades are 1 to 4 and 6 to 12. The blade of which the ID is 5 is lostand is not included in the 3D data.

The second 3D shape 3D142 includes twelve blades. The IDs of the twelveblades are 1 to 12.

After Step S204, the conversion unit 54 converts the 3D coordinatesystem of the 3D data and the 3D coordinate system of the reference datainto a common coordinate system by using the conversion parametergenerated in Step S204. In other words, the conversion unit 54 convertsthe 3D data and the reference data into 3D data in the common coordinatesystem (Step S205). The position, the posture, and the scale of the 3Dcoordinate system of the reference data are not changed and theposition, the posture, and the scale of the 3D coordinate system of the3D data are changed in a case in which the 3D coordinate system of thereference data is used as the common coordinate system.

After Step S205, the display control unit 56 displays an image of a 3Dshape of both the 3D data and the reference data on the display unit 71.At this time, the display control unit 56 controls the state of theimage so that a user can distinguish a region corresponding to the 3Ddata and a region not corresponding to the 3D data from each other (StepS206). When Step S206 is executed, the processing shown in FIG. 49 iscompleted.

FIG. 52 shows an example of an image displayed on the display unit 71.The same parts as those shown in FIG. 51 will not be described. Thedisplay control unit 56 displays an image IMG142 on the display unit 71.The image IMG142 includes a region R143. An image of a 3D shape 3D141and a 3D shape 3D142 is displayed in the region R143.

For example, the display control unit 56 displays the blades having theIDs of 1 to 4 and 6 to 12 in a first color and displays the blade havingthe ID of 5 in a second color different from the first color. The bladehaving the ID of 5 is not included in the 3D data. A user can checkblades included in the reference data and blades not included in thereference data. In other words, the user can check blades of thereference data corresponding to the 3D data and blades of the referencedata not corresponding to the 3D data.

The display control unit 56 may execute processing for highlighting aregion corresponding to the 3D data or a region not corresponding to the3D data. A user may be notified of the position of the regioncorresponding to the 3D data and the position of the region notcorresponding to the 3D data by using voice. As long as the user candistinguish the region corresponding to the 3D data and the region notcorresponding to the 3D data from each other, a method of notifying theuser of a region of the 3D data is not limited to the above-describedexamples.

The method in the sixth embodiment may be applied to an inspection of acombustion chamber of a gas turbine or an inspection of a heat exchangerof the gas turbine. FIG. 53 shows a configuration of a heat exchangerHE150. The heat exchanger HE150 includes two or more heat exchange tubesHT150. In general, an ID is allocated to each of two or more heatexchange tubes in reference data.

An ID may be allocated to 3D data of each heat exchange tube. The imagedisplay device 50 may execute similar processing to that shown in FIG.49 and FIG. 50 . In this way, the image display device 50 may cause theposition, the posture, and the scale of the 3D data of each heatexchange tube to match the position, the posture, and the scale of aheat exchange tube of the reference data, respectively.

In the sixth embodiment, the image display device 50 can cause a user torecognize a region that has been inspected and a region that has notbeen inspected by using the 3D data and the reference data. Compared toa case in which only the 3D data are used, the user can easilydistinguish the region that has been inspected and the region that hasnot been inspected from each other. Therefore, the quality of a reportmade by an inspector is improved.

A manager of an inspection division or a member of a client who requestsan inspection approves the inspection. In general, an approver of theinspection is not very familiar with an endoscopic inspection. Theapprover can objectively determine whether the inspection has beencorrectly performed by checking a region that has been inspected and aregion that has not been inspected. Therefore, reliability of theinspection is secured.

While preferred embodiments of the invention have been described andshown above, it should be understood that these are examples of theinvention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. A three-dimensional image display method,comprising: a first acquisition step in which a processor connects to arecording medium storing first three-dimensional data of a subject andsecond three-dimensional data of the subject and acquires the firstthree-dimensional data from the recording medium, wherein the firstthree-dimensional data include three-dimensional coordinates defined ina first three-dimensional coordinate system, wherein the secondthree-dimensional data include three-dimensional coordinates defined ina second three-dimensional coordinate system different from the firstthree-dimensional coordinate system, and wherein at least part of aregion of the subject corresponding to the first three-dimensional datais different from at least part of a region of the subject correspondingto the second three-dimensional data; a second acquisition step in whichthe processor connects to the recording medium and acquires the secondthree-dimensional data from the recording medium; a conversion step inwhich the processor converts the first three-dimensional coordinatesystem and the second three-dimensional coordinate system into athree-dimensional common coordinate system on the basis of structureinformation related to a geometric structure of the subject, wherein thestructure information is generated without using the firstthree-dimensional data or the second three-dimensional data; and adisplay step in which the processor displays an image of the firstthree-dimensional data in the common coordinate system and an image ofthe second three-dimensional data in the common coordinate system on adisplay.
 2. The three-dimensional image display method according toclaim 1, wherein the first three-dimensional data are generated by usingtwo or more first images acquired at two or more different viewpoints,wherein the second three-dimensional data are generated by using two ormore second images acquired at two or more different viewpoints, andwherein at least one of the two or more first images and at least one ofthe two or more second images are different from each other.
 3. Thethree-dimensional image display method according to claim 2, whereineach of the two or more first images and each of the two or more secondimages include time information, and wherein the processor converts thefirst three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system onthe basis of the time information in the conversion step.
 4. Thethree-dimensional image display method according to claim 2, wherein thetwo or more first images are included in two or more images included ina first video, and wherein the two or more second images are included intwo or more images included in a second video that is the same as ordifferent from the first video.
 5. The three-dimensional image displaymethod according to claim 4, wherein the two or more first images andthe two or more second images are included in the same video file. 6.The three-dimensional image display method according to claim 5, furthercomprising a calculation step in which the processor calculates aposition of a lost region on the basis of the number of the two or morefirst images, the number of the two or more second images, and thenumber of third images, wherein the third images are temporally disposedbetween a set of the two or more first images and a set of the two ormore second images in the video file, wherein the lost region is aregion of the subject different from any one of a first region of thesubject and a second region of the subject, wherein the first regioncorresponds to the three-dimensional coordinates included in the firstthree-dimensional data, wherein the second region corresponds to thethree-dimensional coordinates included in the second three-dimensionaldata, and wherein the processor converts the first three-dimensionalcoordinate system and the second three-dimensional coordinate systeminto the common coordinate system on the basis of the position of thelost region in the conversion step.
 7. The three-dimensional imagedisplay method according to claim 6, wherein the processor calculates ashape of the lost region in the calculation step, and wherein theprocessor converts the first three-dimensional coordinate system and thesecond three-dimensional coordinate system into the common coordinatesystem on the basis of the shape of the lost region in the conversionstep.
 8. The three-dimensional image display method according to claim2, wherein the structure information indicates two or more positions atwhich a distal end of a movable insertion unit capable of being insertedinside an object having the subject is sequentially disposed.
 9. Thethree-dimensional image display method according to claim 8, wherein thestructure information includes first position information and secondposition information, wherein the first position information indicatestwo or more positions at which the distal end is sequentially disposedin order to acquire the two or more first images, wherein the secondposition information indicates two or more positions at which the distalend is sequentially disposed in order to acquire the two or more secondimages, wherein the three-dimensional image display method furthercomprises a generation step in which the processor generates a positionconversion parameter and a posture conversion parameter used forconverting the first three-dimensional coordinate system and the secondthree-dimensional coordinate system into the common coordinate system onthe basis of the first position information and the second positioninformation, and wherein the processor converts the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system by using theposition conversion parameter and the posture conversion parameter inthe conversion step.
 10. The three-dimensional image display methodaccording to claim 2, wherein the processor converts the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system such that a firstregion of the subject and a second region of the subject are connectedtogether in the conversion step, wherein the first region corresponds tothe three-dimensional coordinates included in the firstthree-dimensional data, wherein the second region corresponds to thethree-dimensional coordinates included in the second three-dimensionaldata, and wherein the processor displays information indicatingpositions of the first region and the second region on the display inthe display step.
 11. The three-dimensional image display methodaccording to claim 10, wherein the processor displays informationindicating accuracy of connection between the first region and thesecond region on the display in the display step.
 12. Thethree-dimensional image display method according to claim 2, wherein thetwo or more first images and the two or more second images are generatedby an endoscope.
 13. The three-dimensional image display methodaccording to claim 2, wherein the two or more first images and the twoor more second images are generated on the basis of an optical image ofthe subject acquired through a single-eye optical system.
 14. Thethree-dimensional image display method according to claim 2, furthercomprising a generation step in which the processor generates a scaleconversion parameter used for correcting at least one of a scale of athree-dimensional shape indicated by the first three-dimensional dataand a scale of a three-dimensional shape indicated by the secondthree-dimensional data, wherein the processor converts the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system by using the scaleconversion parameter in the conversion step.
 15. The three-dimensionalimage display method according to claim 2, wherein the structureinformation is configured as design data including a design value of thegeometric structure or is configured as three-dimensional data differentfrom any of the first three-dimensional data and the secondthree-dimensional data.
 16. The three-dimensional image display methodaccording to claim 2, wherein the structure information is generated onthe basis of data output from a sensor.
 17. The three-dimensional imagedisplay method according to claim 2, wherein the two or more firstimages and the two or more second images are generated on the basis ofan optical image of the subject acquired by an insertion unit, whereinthe insertion unit is capable of being inserted inside an object havingthe subject and is bendable, and wherein the structure information isgenerated on the basis of information indicating a bending direction anda bending amount of the insertion unit.
 18. The three-dimensional imagedisplay method according to claim 2, further comprising a generationstep in which the processor generates a position conversion parameterand a posture conversion parameter used for converting the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system on the basis of thestructure information, wherein the processor converts the firstthree-dimensional coordinate system and the second three-dimensionalcoordinate system into the common coordinate system by using theposition conversion parameter and the posture conversion parameter inthe conversion step.
 19. The three-dimensional image display methodaccording to claim 2, further comprising an adjustment step in which theprocessor adjusts at least one of a position and a posture of at leastone of the first three-dimensional data and the second three-dimensionaldata in the common coordinate system.
 20. The three-dimensional imagedisplay method according to claim 2, wherein the subject includes two ormore objects, and wherein the structure information indicates positionsat which the two or more objects are disposed.
 21. A three-dimensionalimage display device, comprising a processor configured to: connect to arecording medium storing first three-dimensional data of a subject andsecond three-dimensional data of the subject, wherein the firstthree-dimensional data include three-dimensional coordinates defined ina first three-dimensional coordinate system, wherein the secondthree-dimensional data include three-dimensional coordinates defined ina second three-dimensional coordinate system different from the firstthree-dimensional coordinate system, and wherein at least part of aregion of the subject corresponding to the first three-dimensional datais different from at least part of a region of the subject correspondingto the second three-dimensional data; acquire the firstthree-dimensional data and the second three-dimensional data from therecording medium; convert the first three-dimensional coordinate systemand the second three-dimensional coordinate system into athree-dimensional common coordinate system on the basis of structureinformation related to a geometric structure of the subject, wherein thestructure information is generated without using the firstthree-dimensional data or the second three-dimensional data; and displayan image of the first three-dimensional data in the common coordinatesystem and an image of the second three-dimensional data in the commoncoordinate system on a display.
 22. A non-transitory computer-readablerecording medium storing a program causing a computer to execute: afirst acquisition step in which the computer connects to a recordingmedium storing first three-dimensional data of a subject and secondthree-dimensional data of the subject and acquires the firstthree-dimensional data from the recording medium, wherein the firstthree-dimensional data include three-dimensional coordinates defined ina first three-dimensional coordinate system, wherein the secondthree-dimensional data include three-dimensional coordinates defined ina second three-dimensional coordinate system different from the firstthree-dimensional coordinate system, and wherein at least part of aregion of the subject corresponding to the first three-dimensional datais different from at least part of a region of the subject correspondingto the second three-dimensional data; a second acquisition step in whichthe computer connects to the recording medium and acquires the secondthree-dimensional data from the recording medium; a conversion step inwhich the computer converts the first three-dimensional coordinatesystem and the second three-dimensional coordinate system into athree-dimensional common coordinate system on the basis of structureinformation related to a geometric structure of the subject, wherein thestructure information is generated without using the firstthree-dimensional data or the second three-dimensional data; and adisplay step in which the computer displays an image of the firstthree-dimensional data in the common coordinate system and an image ofthe second three-dimensional data in the common coordinate system on adisplay.